Re: StartCom inclusion request: next steps
Dear Inigo, On 14/09/17 09:49, Gervase Markham wrote: > The Mozilla CA Certificates team has been considering what the > appropriate next steps are for the inclusion request from the CA > "StartCom".[0] As readers will know, this CA has previously been removed > from trust[1], and so a re-application obviously involves particular > scrutiny. In addition, several questions have been raised about the way > in which the new StartCom PKI has been operated technically[2]. This is > a proposal for the way forward, on which comments are invited. A month ago we posted the above re: StartCom, in what was officially a request for comments. The ensuing discussion did not lead us to conclude that anything we were asking for was inappropriate. So we want to formally confirm that it is indeed Mozilla's requirement that StartCom begin again with a new-new hierarchy and do the other things outlined in the original post before we will further consider a root inclusion request from StartCom. Gerv ___ dev-security-policy mailing list dev-security-policy@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-security-policy
Re: FW: StartCom inclusion request: next steps
On Tue, Sep 19, 2017 at 3:09 PM, Nick Lamb via dev-security-policy wrote: > I have no doubt that this was obvious to people who have worked for a public > CA, but it wasn't obvious to me, so thank you for answering. I think these > answers give us good reason to be confident that a cross-signed certificate > in this situation would not be available to either end subscribers or > StartCom unless/ until the CA which cross-signed it wanted that to happen. > > It might still make sense for Mozilla to clarify that this isn't a good idea, > or even outright forbid it anyway, but I agree with your perspective that > this seemed permissible under the rules as you understood them and wasn't > obviously unreasonable. I'm pretty sure it's already forbidden, since policy version 2.5 anyway (has effective date after the Certinomis shenanigans though): "The CA with a certificate included in Mozilla’s root program MUST disclose this information within a week of certificate creation, and before any such subordinate CA is allowed to issue certificates." 2.5 added the "within a week of certificate creation" [1] . "Creation" vs "My Safe", "Creation" wins. :-) [1] https://github.com/mozilla/pkipolicy/commit/b7d1b6c04458114fbe73fa3f146ad401235c2a1b ___ dev-security-policy mailing list dev-security-policy@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-security-policy
Re: FW: StartCom inclusion request: next steps
On Tuesday, 19 September 2017 15:46:09 UTC+1, Franck Leroy wrote: > 1/ When we use our root, we produce a key ceremony report. > 2/ The signature value doesn’t appears in the report so it is not possible to > reproduce the certificate. > 3/ My safe is in a closet which I don’t have the key, so I have to ask my > manager to open it, then I can open my safe this my key. Thanks Franck, I have no doubt that this was obvious to people who have worked for a public CA, but it wasn't obvious to me, so thank you for answering. I think these answers give us good reason to be confident that a cross-signed certificate in this situation would not be available to either end subscribers or StartCom unless/ until the CA which cross-signed it wanted that to happen. It might still make sense for Mozilla to clarify that this isn't a good idea, or even outright forbid it anyway, but I agree with your perspective that this seemed permissible under the rules as you understood them and wasn't obviously unreasonable. ___ dev-security-policy mailing list dev-security-policy@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-security-policy
Re: FW: StartCom inclusion request: next steps
On Tuesday, September 19, 2017 at 3:46:09 PM UTC+1, Franck Leroy wrote: > Le lundi 18 septembre 2017 17:28:44 UTC+2, Nick Lamb a écrit : > > On Monday, 18 September 2017 15:50:16 UTC+1, Franck Leroy wrote: > > > This control that StartCom was not allowed to use our path was technical > > > in place by the fact that I was the only one to have the intermediate > > > cross signed certificates, stored (retained) in my personal safe. > > > > I see. Three (groups of) questions as someone who does not operate a public > > CA: > > > > When the cross signature certificate was signed did this result in some > > sort of auditable record of the signing? A paper trial, or its electronic > > equivalent - so that any audit team would be aware that the certificate > > existed, regardless of whether they were present when it was created ? > > > > (If so) Was this record inadequate to reproduce the certificate itself, for > > example just consisting of a serial number and other facts ? > > > > Many important functions of a CA are protected by "no lone zone" type > > practices, but would it be possible for you to retrieve the certificate > > from this safe on your own, without oversight by other employees ? > > > > I suspect all the above questions have answers that would be obvious to me > > if I had worked for a public CA but I hope you will humour me with answers > > anyway. > > Hello > > You are right, answers are quite obvious, but I don’t understand the purpose > of your questions (may be I'm lost in the translation...) > > 1/ When we use our root, we produce a key ceremony report. > 2/ The signature value doesn’t appears in the report so it is not possible to > reproduce the certificate. > 3/ My safe is in a closet which I don’t have the key, so I have to ask my > manager to open it, then I can open my safe this my key. > > These are standard practices, and it changes nothing on the fact that we > cross signed in April and send the certificate to StartCom in August and that > cannot be mathematically proven, just my declaration. > > > Franck Hi Franck, Could you provide us with a copy of this key ceremony report after removing confidential security information which is not relevant? James ___ dev-security-policy mailing list dev-security-policy@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-security-policy
Re: FW: StartCom inclusion request: next steps
Le lundi 18 septembre 2017 17:28:44 UTC+2, Nick Lamb a écrit : > On Monday, 18 September 2017 15:50:16 UTC+1, Franck Leroy wrote: > > This control that StartCom was not allowed to use our path was technical in > > place by the fact that I was the only one to have the intermediate cross > > signed certificates, stored (retained) in my personal safe. > > I see. Three (groups of) questions as someone who does not operate a public > CA: > > When the cross signature certificate was signed did this result in some sort > of auditable record of the signing? A paper trial, or its electronic > equivalent - so that any audit team would be aware that the certificate > existed, regardless of whether they were present when it was created ? > > (If so) Was this record inadequate to reproduce the certificate itself, for > example just consisting of a serial number and other facts ? > > Many important functions of a CA are protected by "no lone zone" type > practices, but would it be possible for you to retrieve the certificate from > this safe on your own, without oversight by other employees ? > > I suspect all the above questions have answers that would be obvious to me if > I had worked for a public CA but I hope you will humour me with answers > anyway. Hello You are right, answers are quite obvious, but I don’t understand the purpose of your questions (may be I'm lost in the translation...) 1/ When we use our root, we produce a key ceremony report. 2/ The signature value doesn’t appears in the report so it is not possible to reproduce the certificate. 3/ My safe is in a closet which I don’t have the key, so I have to ask my manager to open it, then I can open my safe this my key. These are standard practices, and it changes nothing on the fact that we cross signed in April and send the certificate to StartCom in August and that cannot be mathematically proven, just my declaration. Franck ___ dev-security-policy mailing list dev-security-policy@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-security-policy
RE: FW: StartCom inclusion request: next steps
Hi Gerv > > But once the cross-signed cert is publicly available (and it is; it's in CT, > however it got there), all of those certificates become trusted (or > potentially > trusted, if the owner reconfigures their webserver to serve the intermediate, > or if Firefox has already encountered it in the current browsing session). > True > > This is something I don´t understand. Why do you say the audits I > > presented don´t meet the BRs? Because of the findings? The auditors > > indicate those were fixed > > I don't believe there's a formal way for an auditor to bindingly say "by the > way, the problems we found have since been fixed" in an audit report. Not bindingly, we provided a CAP (Corrective Action Plan) for all those findings indicated what we did to fix them and provided the evidences and what we were going to do for those that couldn´t be fixed before receiving the report. > But to help me understand: exactly what statement on what page of your audit > report(s) are you referring to here? It´s in a section called "other questions" in which they say "Startcom has developed a plan of corrective actions with the objective of solving the identified exceptions, having been implemented the majority of these actions". > > > About the remediation steps, well, I answered the bug about it providing all > the info and yes, you haven´t answered yet nor to approve nor to deny. > > Right. So why are you proceeding? > > You might reasonably complain it's taken us a while to respond to that > comment about the steps. Yes, it has. The Mozilla inclusion process is slow. > :-( Well, because I wanted to speed up the process if possible. We did everything what was requested and replied the bug. And also applied for the inclussion and none said nothing about it. Kathleen told me that it was going to be slow because the queue was long so I was waiting, no problem, but didn´t know that need to ask permission for applying. > > >>> In fact, recently, I asked for permission to use the Certinomis > >>> cross-signed > >> certificates and have no response. I don´t know if this is an > >> administrative silence which may allow me to use it but until having > >> a clear direction we haven´t used it. > >> > >> Can you remind me how you asked and when? > > > > It was in an email of sept 4th, titled "StartCom communication" in > > which at the end of the long email I asked for feedback to use the > > cross-signed certificates and give additional explanations > > I have no record of any email with that title, or any email from you between > 15th August ("Re: Problem Reporting Mechanism") and 11th September ("Re: > Remove old Startcom roots from NSS"). Where did you send it? I sent it to the m.d.s.p list and got a reply from Andrew Ayer almost inmediately. > > Gerv smime.p7s Description: S/MIME cryptographic signature ___ dev-security-policy mailing list dev-security-policy@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-security-policy
Re: FW: StartCom inclusion request: next steps
Hi Inigo, On 15/09/17 17:30, Inigo Barreira wrote: > There wasn´t a lack of integrity and monitoring, of course not. All PKI logs > were and are signed, it´s just the auditors wanted to add the integrity to > other systems which is not so clear that should have this enabled. For > example, if you want to archive database information for not managing a big > one, the integrity of the logs could be a problem when trying to "move" to an > archive system. I had some discussions about the "scope" of the integrity. > Regarding the monitoring, well, we monitor many things, in both data centers, > 24x7, etc. For this specific issue, it´s true that we didn´t have it > automatically but manually, but well, and we implement a solution, but this > is not a lack of monitoring. I think the audits are to correct and improve > the systems and don´t think any CA at the first time had everything correct. > So, for example, I thought this finding was good because made us improve. > >> Repairing them afterwards does not remove the uncertainty. > > Well, then any issue that you could find, even repaired or fixed, does not > provide you any security and hence you should not trust anyone. Not so. There is particular concern about issues with auditing and monitoring. For other issues, you can check the logs to see whether a particular bug was abused. If the auditing or monitoring is broken or inadequate, you can't tell what happened. >> I may have made a mistake here. I was under the impression that you had >> told me that your new hierarchy, cross-signed by Certnomis, had issued >> 50,000 certificates. Did I misunderstand? If so, my apologies. > > No, or not totally. We have issued those certs but not cross-signed by > Certinomis because we didn´t have the cross-signed certificates so, all of > them were issued under the new startcom hierarchy But once the cross-signed cert is publicly available (and it is; it's in CT, however it got there), all of those certificates become trusted (or potentially trusted, if the owner reconfigures their webserver to serve the intermediate, or if Firefox has already encountered it in the current browsing session). > This is something I don´t understand. Why do you say the audits I presented > don´t meet the BRs? Because of the findings? The auditors indicate those were > fixed I don't believe there's a formal way for an auditor to bindingly say "by the way, the problems we found have since been fixed" in an audit report. But to help me understand: exactly what statement on what page of your audit report(s) are you referring to here? > About the remediation steps, well, I answered the bug about it providing all > the info and yes, you haven´t answered yet nor to approve nor to deny. Right. So why are you proceeding? You might reasonably complain it's taken us a while to respond to that comment about the steps. Yes, it has. The Mozilla inclusion process is slow. :-( >>> In fact, recently, I asked for permission to use the Certinomis cross-signed >> certificates and have no response. I don´t know if this is an administrative >> silence which may allow me to use it but until having a clear direction we >> haven´t used it. >> >> Can you remind me how you asked and when? > > It was in an email of sept 4th, titled "StartCom communication" in which at > the end of the long email I asked for feedback to use the cross-signed > certificates and give additional explanations I have no record of any email with that title, or any email from you between 15th August ("Re: Problem Reporting Mechanism") and 11th September ("Re: Remove old Startcom roots from NSS"). Where did you send it? Gerv ___ dev-security-policy mailing list dev-security-policy@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-security-policy
Re: FW: StartCom inclusion request: next steps
Hi Franck, On 18/09/17 15:49, Franck Leroy wrote: > Our understanding in April was that as long as StartCom is not > allowed by Certinomis to issue EE certs, the disclosure was not > mandated immediately. I think that we need to establish a timeline of the exact events involved here. But I would say that it seems to me that Startcom _were_ issuing EE certs at that time, from the part of their hierarchy that you had cross-signed. In what way was Certnomis forbidding them from doing so? My understanding is that your answer to this question is... > This control that StartCom was not allowed to use our path was > technical in place by the fact that I was the only one to have the > intermediate cross signed certificates, stored (retained) in my > personal safe. that you had not given Startcom a copy of the cross-sign. However, leaving aside for the moment the reasonable question about how such an assertion can be audited, the point is that once the certificate _does_ become public, all of the existing certificates immediately become publicly trusted. Wouldn't you agree? Gerv ___ dev-security-policy mailing list dev-security-policy@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-security-policy
Re: FW: StartCom inclusion request: next steps
On 15/09/17 15:35, Inigo Barreira wrote: > No, those weren´t tests. We allowed the use of curves permitted by the BRs > but this issue came up in the mozilla policy (I think Arkadiusz posted) and I > also asked about it in the last CABF F2F (I asked Ryan about it) and then, > with that outcome and as the browsers didn´t accept them, we revoked and then > not allow the issuance. I think the discussion is still active (i.e. the use > of P-521). Unless Mozilla says otherwise (which we do occasionally), you should not be basing your practice on what we are discussing, or even on draft versions of the policy, but on the published version - which clearly prohibits P-521. Gerv ___ dev-security-policy mailing list dev-security-policy@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-security-policy
Re: FW: StartCom inclusion request: next steps
On Monday, 18 September 2017 15:50:16 UTC+1, Franck Leroy wrote: > This control that StartCom was not allowed to use our path was technical in > place by the fact that I was the only one to have the intermediate cross > signed certificates, stored (retained) in my personal safe. I see. Three (groups of) questions as someone who does not operate a public CA: When the cross signature certificate was signed did this result in some sort of auditable record of the signing? A paper trial, or its electronic equivalent - so that any audit team would be aware that the certificate existed, regardless of whether they were present when it was created ? (If so) Was this record inadequate to reproduce the certificate itself, for example just consisting of a serial number and other facts ? Many important functions of a CA are protected by "no lone zone" type practices, but would it be possible for you to retrieve the certificate from this safe on your own, without oversight by other employees ? I suspect all the above questions have answers that would be obvious to me if I had worked for a public CA but I hope you will humour me with answers anyway. ___ dev-security-policy mailing list dev-security-policy@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-security-policy
Re: FW: StartCom inclusion request: next steps
Le lundi 18 septembre 2017 14:52:27 UTC+2, Ryan Sleevi a écrit : > On Mon, Sep 18, 2017 at 8:12 AM, Inigo Barreira <> > wrote: > Then they misissued a CA certificate and failed to disclose it, and we > should start an incident report into it. Hello In April 2017 the mozilla policy in force (v2.4) stated: “The CA with a certificate included in Mozilla’s CA Certificate Program MUST disclose this information before any such subordinate CA is allowed to issue certificates.” Our understanding in April was that as long as StartCom is not allowed by Certinomis to issue EE certs, the disclosure was not mandated immediately. This control that StartCom was not allowed to use our path was technical in place by the fact that I was the only one to have the intermediate cross signed certificates, stored (retained) in my personal safe. As soon as Certinomis has authorized StartCom to use the path to our root, I disclosed the certificates with the audit reports in the CCADB, and send the certificates to Inigo. May be I misunderstood the Mozilla requirements v2.4, and as I already said in previous post, I do apologize for it. But it was not my intention not to enforce the policy; I personally took care that StartCom could not be able to use the path to our root until a full BR audit assessment report was provided. Regards Franck Leroy ___ dev-security-policy mailing list dev-security-policy@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-security-policy
Re: FW: StartCom inclusion request: next steps
On Mon, Sep 18, 2017 at 8:12 AM, Inigo Barreira wrote: > > We are not seeking to identify personal blame. We are seeking to > understand what, if any, improvements have been made to address such > issues. In reading this thread, I have difficulty finding any discussion > about the steps that StartCom has proposed to improve its awareness of the > expectations placed upon it as a potential participant in the Mozilla > store. Regardless of who bears responsibility for that, the absence of a > robust process - and, unfortunately, the absence of a deep understanding - > does mean that the restablishing of trust can present a significant risk to > the community. > > > > I think I´ve posted everything we did to improve our systems. I replied to > every error posted in the crt.sh explaining what happened and what we did > to fix it for not having the same issue again, but will try to recap here > again. > > > > - Test certificates. We issued test certificates in production > to test the CT behaviour. After the checking those certs were revoked, > within minutes. This was due to an incorrect configuration in the EJBCA > roles that was changed and updated accordingly for not allowing anyone to > issue certs from the EJBCA directly > > - Use of unallowed curves. We issued certificates with P-521 > which is not allowed by Mozilla. We revoked all those certs and configure > the system to not allow it. This remediation was put into production on > mid-july not issuing certs with that curve. > > - RSA parameter not included. We issued one certificate which no > RSA parameter included. We revoked that certificate and started an > investigation. The EJBCA system didn´t check the keys, concretely for this > issue. We developed a solution to check the CSR files properly before > sending to sign > > - Country code not allowed. We issued one certificate with > country code ZR for Zaire, which does not exist officially. We revoked the > cert and checked our internal country code database with the ISO one. We > made the correspondent changes. The cert was reissued with the right code > representing the Democratic Republic of Congo. > > > > Furthermore, we have added x509lint and cablint to our issuance process. > We have integrated crt.sh tool into our CMS system. We have developed a CSR > checking tool. We have updated the EJBCA system to the latest patch, > 6.0.9.5 which also came with a Key (RSA and ECC) validator, and we are also > willing to integrate the zlint once is getting more stable. We have applied > all these tools and we are not misissuing certificates. > Unfortunately, I am not sure how to more effectively communicate that this pattern of issues indicates an organization failure in the review of, application of, and implementation of the Baseline Requirements. Through both coding practices and issuing practices, security and compliance are not being responded to as systemic objectives, but rather as 'one offs', giving the impression of 'whack-a-mole' and ad-hoc response. For example, the country code failure indicates a more deeper failing - did you misunderstand the BRs? Were they not reviewed? Was the code simply not implemented correctly? With the RSA parameters, it similarly indicates a lack of attention. I greatly appreciate the use of and deployment of x509lint and cablint, but those merely offer technical checking that, as an aspiring trusted root CA, you should have already been implementing - whether your own or using those available COTS. The continued approach to issue-and-revoke rather than holistically review the practices and take every step possible to ensure compliance - particularly at a CA that was previously distrusted due to non-compliance - is a particularly egregious oversight that hasn't been responded to. In every response, it still feels as if you're suggesting these are one-offs and coding errors, when the concern being raised is how deeply indicative they are of systemic failures from top to bottom, from policy to technology, from oversight to implementation. Rather than demonstrate how beyond reproach StartCom is, it feels like an excessive emphasis is being put on the ineffective revocation of these certificates, while ignoring the issues that lead to them being issued in the first place - both from policy and from code. It´s not like disagreements, but the example was about a root certificate > private key in a USB stick, so IMO that example starts with a very > problematic issue, because it´s about a root private key and in a USB stick > left in the table, while the issues Startcom did was about end entity > certificates, and nothing related to private keys and not in the root, > that´s what I meant with “quality”. > I understand what you meant. I'm suggesting the community has a perception that the issues StartCom is presently being faced with is as egregious and as serious. I understand you disagree it's as significant. The suggestion is
Re: StartCom inclusion request: next steps
On Monday, September 18, 2017 at 11:38:57 AM UTC+1, Inigo Barreira wrote: > > > > I want to give you some words from one of the "community side" (this is a > > personal opinion and may vary from other opinions inside the community). > > > > Trust is not something that you get, it is something that you earn. > > True > > > StartCom was distrusted because of serious issues with their old PKI and now > > had the chance to restart - there are serious issues again. I don't think > > that > > the "community" wants rogue CAs on its list just because they restarted with > > new certificates. > > > > - The fact that you were cross-signed by Certnomis before you had valid > > WebTrust Audits and the permission to issue trusted certificates again and > > that the only thing which prevented you from using the trust path is a > > PUBLIC > > certificate? Is the only thing that prevents me from entering your > > datacenter > > a sign which tells me not to do so and the fact that you did not tell me > > where > > your datacenter is located? > > > > - Startcom operates/operated multiple CT Log Server itself. There is > > absolutely > > no reason to use trusted certificates for testing purposes if he does have a > > testing infrastructure. It would be easy for you to add one of your testing > > roots to your CT Logs and then test your CT behaviour. I don't think that > > Googles CTs are different from your own ones. Though your certificates might > > not have been trusted at that time, they would be now and as Gerv said, test > > certificates are not allowed. If you did not care about compliance at that > > time, why should you care about it now? > > Those certificates were not trusted at that time and can´t be now because > they were revoked within minutes. > > > > > - There is a reason why Best Practices are called best practices. Why did > > you > > reuse your key in root and intermediate certificates? > > Because there is nomoney for additional HSMs? Because you don't know how to > > generate new > > keys? An explanation would be great. > > A new thread has started about this. It´s not forbidden. > > > > > - P-521 are forbidden by Mozilla. Even if there is a discussion to change > > this it > > does not allow you to take that as a permission to test it. The fact that > > these > > certificates were reported as unrevoked at the time of reporting (as far as > > I > > remember) does imply that you do not monitor your certificate issuances for > > policy compliance at all. What do you do to ensure that all of your > > certificates > > are compliant with all requirements at all times? > > At the time of application, the certificates were revoked and countermeasures > set and since then there´s no more issues. We have implemented cablint, > crt.sh, ... and some other tools into our issuance process and still trying > to improve much more. > I´m not trying to excuse we had issues but we corrected them. > > > > > - What internal audits have you done to ensure the integrity of your > > systems? > > If something so critically as the permission to issue certificates in EJBCA > > is > > only noticed after you explicitly looked for it, what happens if someone > > removes all of your security mechanisms? You will find that out too after > > you > > misissued thousands of certificates? Quis custodiet ipsos custodes. > > Despite all the terrible systems we have, etc. we haven´t misissued thousands > of certificates, nor hundreds. The issues we had, have been fixed. Those > test certs issued directly from the EJBCA was a mistake, explained many > times. I have nothing to add to what I´ve already said. It was not a good > decission, not a good practice, and it´s forbidden. > > > > > - The incidents with Diginotar should have made clear that secure, well > > audited and hardened code is absolutely necessary as well as reliable logs. > > The fact that these flaws where not found by your internal team and only > > discovered after an external company tested your systems is deeply > > concerning. What have you done now and what will you do to ensure that > > your systems won't be abused? How do you make sure that the code your > > people write in the future is safe and how do you detect security problems > > if > > you were unable to do so at the first time? > > This is a different example, Diginotar was attacked and the attacker was able > to enter in their systems, and this is not what happened with StartCom. As > said, the code that went live is not the same that was audited the first time > and has been improved since then. The audits are just for that, and we will > continue doing yearly security audits to improve our systems. Why not open-source the code on GitHub and let us be the judge of the improvements made to your systems code? Lets encrypt does this and works successfully. > > > > Though I would love to see StartCom up and running again, I have to agree > > with James that a
RE: StartCom inclusion request: next steps
> > I want to give you some words from one of the "community side" (this is a > personal opinion and may vary from other opinions inside the community). > > Trust is not something that you get, it is something that you earn. True > StartCom was distrusted because of serious issues with their old PKI and now > had the chance to restart - there are serious issues again. I don't think that > the "community" wants rogue CAs on its list just because they restarted with > new certificates. > > - The fact that you were cross-signed by Certnomis before you had valid > WebTrust Audits and the permission to issue trusted certificates again and > that the only thing which prevented you from using the trust path is a PUBLIC > certificate? Is the only thing that prevents me from entering your datacenter > a sign which tells me not to do so and the fact that you did not tell me where > your datacenter is located? > > - Startcom operates/operated multiple CT Log Server itself. There is > absolutely > no reason to use trusted certificates for testing purposes if he does have a > testing infrastructure. It would be easy for you to add one of your testing > roots to your CT Logs and then test your CT behaviour. I don't think that > Googles CTs are different from your own ones. Though your certificates might > not have been trusted at that time, they would be now and as Gerv said, test > certificates are not allowed. If you did not care about compliance at that > time, why should you care about it now? Those certificates were not trusted at that time and can´t be now because they were revoked within minutes. > > - There is a reason why Best Practices are called best practices. Why did you > reuse your key in root and intermediate certificates? > Because there is nomoney for additional HSMs? Because you don't know how to > generate new > keys? An explanation would be great. A new thread has started about this. It´s not forbidden. > > - P-521 are forbidden by Mozilla. Even if there is a discussion to change > this it > does not allow you to take that as a permission to test it. The fact that > these > certificates were reported as unrevoked at the time of reporting (as far as I > remember) does imply that you do not monitor your certificate issuances for > policy compliance at all. What do you do to ensure that all of your > certificates > are compliant with all requirements at all times? At the time of application, the certificates were revoked and countermeasures set and since then there´s no more issues. We have implemented cablint, crt.sh, ... and some other tools into our issuance process and still trying to improve much more. I´m not trying to excuse we had issues but we corrected them. > > - What internal audits have you done to ensure the integrity of your systems? > If something so critically as the permission to issue certificates in EJBCA is > only noticed after you explicitly looked for it, what happens if someone > removes all of your security mechanisms? You will find that out too after you > misissued thousands of certificates? Quis custodiet ipsos custodes. Despite all the terrible systems we have, etc. we haven´t misissued thousands of certificates, nor hundreds. The issues we had, have been fixed. Those test certs issued directly from the EJBCA was a mistake, explained many times. I have nothing to add to what I´ve already said. It was not a good decission, not a good practice, and it´s forbidden. > > - The incidents with Diginotar should have made clear that secure, well > audited and hardened code is absolutely necessary as well as reliable logs. > The fact that these flaws where not found by your internal team and only > discovered after an external company tested your systems is deeply > concerning. What have you done now and what will you do to ensure that > your systems won't be abused? How do you make sure that the code your > people write in the future is safe and how do you detect security problems if > you were unable to do so at the first time? This is a different example, Diginotar was attacked and the attacker was able to enter in their systems, and this is not what happened with StartCom. As said, the code that went live is not the same that was audited the first time and has been improved since then. The audits are just for that, and we will continue doing yearly security audits to improve our systems. > > Though I would love to see StartCom up and running again, I have to agree > with James that all of these issues do not enwake trust into you and instead > produce more uncertainties if StartCom is really able to run a PKI itself. > But as > I said before, this is a personal opinion :) > > > Am Freitag, 15. September 2017 16:38:25 UTC+2 schrieb Inigo Barreira: > > > > Yes, you´re right, that was on the table and also suggested by > > > > Mozilla, but the issue was that people from 360 are used to code > > > > in PHP and the old one was in Java and som
Re: FW: StartCom inclusion request: next steps
I didn't understand the original below comment by StartCom very well about the cross-sign, but after Ryan's message I understand it better in retrospect: > On Thu, Sep 14, 2017 at 11:05 AM, Inigo Barreira via dev-security-policy < dev-security-policy@lists.mozilla.org> wrote: > > I´ve never said this. In fact, despite having that cross-signed which were provided to us in july we have never used and provided to any of our customers to build a trusted path. So none of those 5, or the new ones, go with the Certinomis path because none have it. But all those 5 certs are untrusted because we´re not in the Mozilla root, not the new one, and the old one was distrusted. > > In fact, recently, I asked for permission to use the Certinomis cross-signed certificates and have no response. I don´t know if this is an administrative silence which may allow me to use it but until having a clear direction we haven´t used it. So this appears to be saying that "all those 5 certs are untrusted" because StartCom didn't provide the full chain to customers, even though such a chain could be constructed. The cross-signature wasn't published in CT until August 2nd, but that's not any sort of guarantee that the cross-signature wasn't discoverable by other means -- its availability until August 2nd is a function of actions by Certinomis that are not disclosed. The August 2nd date is also after StartCom's actions were being publicly questioned, so it suggests the possibility that the cross-signature would have been kept secret for longer, and was only submitted to CT once scrutiny had increased. Whether the cross-cert was issued before the audit report date is also a mystery, especially if it's possible that either Certinomis or StartCom was operating under the assumption that the cross-signature is irrelevant until "delivered" to customers. StartCom has remarked several times in this thread that they are being treated unfairly, but I can think of at least one comparison to a previous distrust event, which is that one of the more significant (in my opinion) issues with Symantec's now-deprecated PKI is that there existed chains that brought U.S. Federal PKI certificates into being trusted by Mozilla. Those chains were, as far as I know, never delivered proactively to customers, but could easily be constructed by any interested party with sufficient knowledge of the universe of cross-signatures. For example, Qualys' SSL Labs reports would automatically construct those chains for sites using FPKI certs, and let users download the full chain in one click. The threat model here is not what ordinary inexpert customers do, but what opportunities an adversary has available to them among the universe of trusted CAs to obtain certificates. In the Symantec/FPKI case, the problem was that an adversary could easily use an FPKI certificate to intercept connections made by Mozilla products, whether or not Symantec or the FPKI ever advertised or proactively enabled this use case. What made this such a big issue, in addition to the scope of the technical impact, is that the issue was not noticed or elevated for years, during which multiple "generations" of cross-signs had been issued and expired. It brought Symantec's ability to understand their own PKI into serious question. So I think the biggest issue here is not so much the technical impact, but that StartCom was communicating inaccurate information to Mozilla. The certs were publicly trusted by Certinomis, whether the cross-signature was delivered to StartCom or to customers or to no one. While presumably this inaccuracy was unintentional, it was enough to cause Gerv to express public confusion and doubt about whether the certificates were part of the cross-signed hierarchy. It also reflects a potentially dangerous difference of perspective between StartCom and root stores in how StartCom evaluates the trust and impact of the certificates they issue. For a CA that has been operating for as long as StartCom has, I think it's fair to describe this as concerning. I also think that Certinomis, whose cross-signing practices are now being scrutinized, should proactively post to this list with a timeline of its own actions during this process, so that their actions can be understood in the context of StartCom's. -- Eric ___ dev-security-policy mailing list dev-security-policy@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-security-policy
Re: StartCom inclusion request: next steps
I want to give you some words from one of the "community side" (this is a personal opinion and may vary from other opinions inside the community). Trust is not something that you get, it is something that you earn. StartCom was distrusted because of serious issues with their old PKI and now had the chance to restart - there are serious issues again. I don't think that the "community" wants rogue CAs on its list just because they restarted with new certificates. - The fact that you were cross-signed by Certnomis before you had valid WebTrust Audits and the permission to issue trusted certificates again and that the only thing which prevented you from using the trust path is a PUBLIC certificate? Is the only thing that prevents me from entering your datacenter a sign which tells me not to do so and the fact that you did not tell me where your datacenter is located? - Startcom operates/operated multiple CT Log Server itself. There is absolutely no reason to use trusted certificates for testing purposes if he does have a testing infrastructure. It would be easy for you to add one of your testing roots to your CT Logs and then test your CT behaviour. I don't think that Googles CTs are different from your own ones. Though your certificates might not have been trusted at that time, they would be now and as Gerv said, test certificates are not allowed. If you did not care about compliance at that time, why should you care about it now? - There is a reason why Best Practices are called best practices. Why did you reuse your key in root and intermediate certificates? Because there is no money for additional HSMs? Because you don't know how to generate new keys? An explanation would be great. - P-521 are forbidden by Mozilla. Even if there is a discussion to change this it does not allow you to take that as a permission to test it. The fact that these certificates were reported as unrevoked at the time of reporting (as far as I remember) does imply that you do not monitor your certificate issuances for policy compliance at all. What do you do to ensure that all of your certificates are compliant with all requirements at all times? - What internal audits have you done to ensure the integrity of your systems? If something so critically as the permission to issue certificates in EJBCA is only noticed after you explicitly looked for it, what happens if someone removes all of your security mechanisms? You will find that out too after you misissued thousands of certificates? Quis custodiet ipsos custodes. - The incidents with Diginotar should have made clear that secure, well audited and hardened code is absolutely necessary as well as reliable logs. The fact that these flaws where not found by your internal team and only discovered after an external company tested your systems is deeply concerning. What have you done now and what will you do to ensure that your systems won't be abused? How do you make sure that the code your people write in the future is safe and how do you detect security problems if you were unable to do so at the first time? Though I would love to see StartCom up and running again, I have to agree with James that all of these issues do not enwake trust into you and instead produce more uncertainties if StartCom is really able to run a PKI itself. But as I said before, this is a personal opinion :) Am Freitag, 15. September 2017 16:38:25 UTC+2 schrieb Inigo Barreira: > > > Yes, you´re right, that was on the table and also suggested by > > > Mozilla, but the issue was that people from 360 are used to code in > > > PHP and the old one was in Java and some other for which they are not > > > so familiar and then was decided to re-write all the code in PHP > > > trying to keep the same functionality. > > > > Given the quality of code produced, > > I don´t think the quality of the code which is in production now is poor or > of bad quality. It wasn´t good initially, that´s true, but not now. > > > it might have been better in hindsight tohire Java experts to work on the > > old codebase. > > That was also on the table. > > > > > > Furthermore, with this decission, we also wanted to let the community > > > know that this is totally a new CA system in all aspects, nothing > > > related to the past, everything from scratch, so new coding, new > > > programming language, new PKI system, infrastructure, etc. hoping this > > > would make the community have a better impression of the new Startcom > > regarding the distrust issue. > > > > "We rewrote everything from scratch" is not actually something which itself > > inspires confidence. > > What I meant, is that we used a new programming language and then recoded. > > In the case of WoSign, it was required of them because > > their old code was clearly terrible and buggy. But the reason the result > > would > > have to be strongly audited (were they to > > reapply) is that new code is riskier than ol
Re: FW: StartCom inclusion request: next steps
On Fri, Sep 15, 2017 at 12:30 PM, Inigo Barreira via dev-security-policy < dev-security-policy@lists.mozilla.org> wrote: > > > > Hi Inigo, > > > > On 14/09/17 16:05, Inigo Barreira wrote: > > > Those tests were done to check the CT behaviour, there was any other > > testing of the new systems, just for the CT. > > > > Is there any reason those tests could not have been done using a parallel > > testing hierarchy (other than the fact that you hadn't set one up)? > > I think I provided the reasons. We were distrusted, not re-applied yet, > those certs lived for minutes, ... Can you point where the Baseline Requirements and/or Root Store policies exclude "certs lived for minutes" from being in scope? Or where revocation absolves a CA of the issuance in the first place? > So, I think these are the reasons. It´s not an excuse and we didn´t expect > this maremágnum. I believe the lack of expectation is precisely why there is significant concern here. CAs are expected to be global stewards of trust, operating beyond reproach - which generally means assuming all things are forbidden unless expressly permitted, and even when it seems they _may_ be permitted, to consider the full implications of the interpretation and to check if there are multiple interpretations. > There´s been long discussions about the harm long term certificates can > make and asked CAs about short term to avoid damages. These certs, that > it´s true was a mistake, only lived for minutes, so I don´t know what else > I can add to this matter. > I think you've added a lot, and while the engagement has been valuable, hopefully this clarifies precisely why these responses have been so deeply concerning towards demonstrating trustworthiness. > > But it is the job of a CA to be aware of browser policies. > > Yes, you´re right. And it was my fault for not have checked in deep this > particular one. > We are not seeking to identify personal blame. We are seeking to understand what, if any, improvements have been made to address such issues. In reading this thread, I have difficulty finding any discussion about the steps that StartCom has proposed to improve its awareness of the expectations placed upon it as a potential participant in the Mozilla store. Regardless of who bears responsibility for that, the absence of a robust process - and, unfortunately, the absence of a deep understanding - does mean that the restablishing of trust can present a significant risk to the community. > > I think lack of monitoring and lack of integrity of logs are serious > issues. > > There wasn´t a lack of integrity and monitoring, of course not. All PKI > logs were and are signed, it´s just the auditors wanted to add the > integrity to other systems which is not so clear that should have this > enabled. For example, if you want to archive database information for not > managing a big one, the integrity of the logs could be a problem when > trying to "move" to an archive system. I had some discussions about the > "scope" of the integrity. I am wholly uncertain how to interpret what you're saying here. > Regarding the monitoring, well, we monitor many things, in both data > centers, 24x7, etc. For this specific issue, it´s true that we didn´t have > it automatically but manually, but well, and we implement a solution, but > this is not a lack of monitoring. I think the audits are to correct and > improve the systems and don´t think any CA at the first time had everything > correct. So, for example, I thought this finding was good because made us > improve. > I agree that a well-executed audit can help a CA identify areas of improvement. However, a well-executed audit can also identify issues of non-compliance or identify issues of risk that the community may find unacceptable, independent of the auditors own assessment. > > Repairing them afterwards does not remove the uncertainty. > > Well, then any issue that you could find, even repaired or fixed, does not > provide you any security and hence you should not trust anyone. > I do not think this demonstrates a positive awareness of the issues being discussed. Again, as CAs are expected to be stewards of global trust, it is expected that CAs seek to both individually improve and rise above 'the minimum' requirements, and to seek ways to improve those minimums. > If you said "I left the root certificate private key on a USB stick on > the desk in my unlocked > > office over the weekend - but it's OK, I've remediated the problem now, > and > > it's back in the safe", that would not remove the uncertainty about > whether > > someone had done something with it in the mean time. > > Well, I don´t think this is the same "quality" of example. > It is useful to know what you think, as it highlights disagreement. However, it's more useful to know why you think this. I would suggest that, for most people examining both the information provided on the issues, the facts, and the discussion so far, it is very
RE: FW: StartCom inclusion request: next steps
> > Hi Inigo, > > On 14/09/17 16:05, Inigo Barreira wrote: > > Those tests were done to check the CT behaviour, there was any other > testing of the new systems, just for the CT. > > Is there any reason those tests could not have been done using a parallel > testing hierarchy (other than the fact that you hadn't set one up)? I think I provided the reasons. We were distrusted, not re-applied yet, those certs lived for minutes, ... So, I think these are the reasons. It´s not an excuse and we didn´t expect this maremágnum. There´s been long discussions about the harm long term certificates can make and asked CAs about short term to avoid damages. These certs, that it´s true was a mistake, only lived for minutes, so I don´t know what else I can add to this matter. > > > Some of these "mis-issuances" were due to some incongruencies between > > the BRs and the Mozilla policy, such as the use of different curves > > (allowed by the BRs but not for some browsers), > > But it is the job of a CA to be aware of browser policies. Yes, you´re right. And it was my fault for not have checked in deep this particular one. > > > As far as I know, the current manager of Startcom has not been previously > accused of deception or bad action. > > No, indeed. There is no accusation that you have been deceptive towards > anyone. > > > Yes, it´s not a policy violation. As explained, this was a problem in the > > EJBCA > with the UTF8 encoding. > > That was the reason for the revocation; it's not the reason for the key reuse. Yes, right, but this is allowed afaik. It´s mentioned in the BRs. > > >> * The WT/BR/EV audits on StartCom's website are significantly > >> qualified, and they include lack of controls on issuance. They should > >> have clean ones done before we permit any inclusion request to proceed. > The qualifications include: > >> > >> - Risk analysis process defined but not implemented > >> - Business continuity plan defined but not implemented > >> - Audit logs not guaranteed to have integrity > >> - Monitoring system cannot detect security-related changes to > >> Certificate Systems > > > > Yes, this is also true. Our webtrust audits have findings but those are not > > so > significant according to the auditors who signed the reports, so I assume the > auditors thought that the system is good enough to have the audit report in > place. > > I think lack of monitoring and lack of integrity of logs are serious issues. There wasn´t a lack of integrity and monitoring, of course not. All PKI logs were and are signed, it´s just the auditors wanted to add the integrity to other systems which is not so clear that should have this enabled. For example, if you want to archive database information for not managing a big one, the integrity of the logs could be a problem when trying to "move" to an archive system. I had some discussions about the "scope" of the integrity. Regarding the monitoring, well, we monitor many things, in both data centers, 24x7, etc. For this specific issue, it´s true that we didn´t have it automatically but manually, but well, and we implement a solution, but this is not a lack of monitoring. I think the audits are to correct and improve the systems and don´t think any CA at the first time had everything correct. So, for example, I thought this finding was good because made us improve. > Repairing them afterwards does not remove the uncertainty. Well, then any issue that you could find, even repaired or fixed, does not provide you any security and hence you should not trust anyone. > If you said "I left the root certificate private key on a USB stick on the > desk in my unlocked > office over the weekend - but it's OK, I've remediated the problem now, and > it's back in the safe", that would not remove the uncertainty about whether > someone had done something with it in the mean time. Well, I don´t think this is the same "quality" of example. > > > I don´t know how these you mention have applied, but I remember lots of > issues regarding Google and the acquisition of the Globalsign roots and how > they proceeded. > > I think "lots of issues" is an overstatement. There was an issue regarding the > scope of their audits which was raised publicly, but it was something that > Google has explicitly sought our approval for in advance. Yes, it´s an overstatement but similar to others. I said that I didn´t know what or how they had applied but you put them as examples and just wanted to indicate that there were discussions, so maybe was not so smooth, but again, I don´t know. > > > Not sure about this. We were distrusted in october 2016, the new system > started to operate in april 2017, which is not related to the old one which > has > been switched off. None of the new certs are trusted in Mozilla Firefox and > we notify our users so by messages in the web and applications. > > I may have made a mistake here. I was under the impression that you had > to
RE: FW: StartCom inclusion request: next steps
Well, the use of P-521 is not compliant to Mozilla but it is to the BRs, so don´t understand what you mean. I think we´re not the only one which has issued certs using P-521 for example, and I assume that all other CAs have tested, established and verified their certs are compliant. We´re seeing everyone made mistakes, mis-issuances, and I think all have tested everything. Regarding the use of these curves, I think even Gerv said in August to leave the policy as it is, the BRs still allows the use. After checking that was “incorrect” we revoked the certs and set the new conditions not allowing them. Since July we don´t use these curves, which is a month before Mozilla decission. Best regards Iñigo Barreira CEO StartCom CA Limited From: Alex Gaynor [mailto:agay...@mozilla.com] Sent: viernes, 15 de septiembre de 2017 16:32 To: Inigo Barreira Cc: Gervase Markham ; James Burton ; mozilla-dev-security-pol...@lists.mozilla.org Subject: Re: FW: StartCom inclusion request: next steps I'm fairly confused by your answers, if the only thing you tested in production was CT, why was the system issuing non-compliant certs? Why did production CT testing come before having established, tested, and verified a compliant certificate profile? Alex On Fri, Sep 15, 2017 at 10:35 AM, Inigo Barreira via dev-security-policy mailto:dev-security-policy@lists.mozilla.org> > wrote: > On 15/09/17 11:01, Inigo Barreira wrote: > > Considering that we were distrusted, that we didn´t reapply for > > inclussion, that CT is only required by Chrome and it´s not included > > in the Mozilla policy (even we were requested that all of our certs > > had to be CT logged) nor required by Firefox, that those certs were > > under our control all the time and lived for some minutes because were > > revoked inmediately, at that time, when we did it, we didn´t expect > > this reaction for sure. > > But surely CT testing is not the only sort of testing you've been doing? Yes, this is the only test we did it in production > E.g. you made some test certificates with different types of ECC curve, which > you then had to revoke some of as against browser policies. No, those weren´t tests. We allowed the use of curves permitted by the BRs but this issue came up in the mozilla policy (I think Arkadiusz posted) and I also asked about it in the last CABF F2F (I asked Ryan about it) and then, with that outcome and as the browsers didn´t accept them, we revoked and then not allow the issuance. I think the discussion is still active (i.e. the use of P-521). > If these had been in a testing hierarchy there would have been no problem. > > CAs have been heavily criticised over the past few years for issuing test > certificates in public hierarchies (see e.g. Symantec). The danger of doing so > should be well known to all CAs by now. Yes, I know. But the only testing we did in production was the one related to the CT. > > Perhaps once a test has been passed and checked in a testing system, and if > the certificates concerned do not violate any policies, it could be repeated > on > a production system to deal with any possible differences between the two. > But starting with the production system is not a good idea. True, but it seems you´re understanding that we have only a production system in which we test everything and this is not the case. Before moving anything into production, we have tested in development and in the QA system. > > Gerv ___ dev-security-policy mailing list dev-security-policy@lists.mozilla.org <mailto:dev-security-policy@lists.mozilla.org> https://lists.mozilla.org/listinfo/dev-security-policy smime.p7s Description: S/MIME cryptographic signature ___ dev-security-policy mailing list dev-security-policy@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-security-policy
RE: StartCom inclusion request: next steps
> > Yes, you´re right, that was on the table and also suggested by > > Mozilla, but the issue was that people from 360 are used to code in > > PHP and the old one was in Java and some other for which they are not > > so familiar and then was decided to re-write all the code in PHP > > trying to keep the same functionality. > > Given the quality of code produced, I don´t think the quality of the code which is in production now is poor or of bad quality. It wasn´t good initially, that´s true, but not now. > it might have been better in hindsight tohire Java experts to work on the old > codebase. That was also on the table. > > > Furthermore, with this decission, we also wanted to let the community > > know that this is totally a new CA system in all aspects, nothing > > related to the past, everything from scratch, so new coding, new > > programming language, new PKI system, infrastructure, etc. hoping this > > would make the community have a better impression of the new Startcom > regarding the distrust issue. > > "We rewrote everything from scratch" is not actually something which itself > inspires confidence. What I meant, is that we used a new programming language and then recoded. In the case of WoSign, it was required of them because > their old code was clearly terrible and buggy. But the reason the result would > have to be strongly audited (were they to > reapply) is that new code is riskier than old, tried-and-tested code. > > I don't know if I ever wrote it down anywhere, but I'm fairly sure that > switching back from the WoSign codebase to the older StartCom codebase > (i.e. reversing the change you made after the purchase) was my suggestion for > how StartCom should proceed after the dis-trust event. Yes, that was your suggestion. > That doesn't mean you are required to take my advice, Yes, I know > but it might have beena hint that I wouldn't consider "hey, we rewrote > everything from scratch!" as > a positive point. Well, we thought that it could be. During the distrust issues, I think Ryan posted some old issues related to the old Startcom code or procedures (long time ago) and then recoding everything was our intent to give a positive answer. As said, the term "from scratch" maybe it´s not appropiate, but in the end this code has been audited. > > Gerv smime.p7s Description: S/MIME cryptographic signature ___ dev-security-policy mailing list dev-security-policy@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-security-policy
Re: FW: StartCom inclusion request: next steps
I'm fairly confused by your answers, if the only thing you tested in production was CT, why was the system issuing non-compliant certs? Why did production CT testing come before having established, tested, and verified a compliant certificate profile? Alex On Fri, Sep 15, 2017 at 10:35 AM, Inigo Barreira via dev-security-policy < dev-security-policy@lists.mozilla.org> wrote: > > On 15/09/17 11:01, Inigo Barreira wrote: > > > Considering that we were distrusted, that we didn´t reapply for > > > inclussion, that CT is only required by Chrome and it´s not included > > > in the Mozilla policy (even we were requested that all of our certs > > > had to be CT logged) nor required by Firefox, that those certs were > > > under our control all the time and lived for some minutes because were > > > revoked inmediately, at that time, when we did it, we didn´t expect > > > this reaction for sure. > > > > But surely CT testing is not the only sort of testing you've been doing? > > Yes, this is the only test we did it in production > > > E.g. you made some test certificates with different types of ECC curve, > which > > you then had to revoke some of as against browser policies. > > No, those weren´t tests. We allowed the use of curves permitted by the BRs > but this issue came up in the mozilla policy (I think Arkadiusz posted) and > I also asked about it in the last CABF F2F (I asked Ryan about it) and > then, with that outcome and as the browsers didn´t accept them, we revoked > and then not allow the issuance. I think the discussion is still active > (i.e. the use of P-521). > > > If these had been in a testing hierarchy there would have been no > problem. > > > > CAs have been heavily criticised over the past few years for issuing test > > certificates in public hierarchies (see e.g. Symantec). The danger of > doing so > > should be well known to all CAs by now. > > Yes, I know. But the only testing we did in production was the one related > to the CT. > > > > Perhaps once a test has been passed and checked in a testing system, and > if > > the certificates concerned do not violate any policies, it could be > repeated on > > a production system to deal with any possible differences between the > two. > > But starting with the production system is not a good idea. > > True, but it seems you´re understanding that we have only a production > system in which we test everything and this is not the case. Before moving > anything into production, we have tested in development and in the QA > system. > > > > Gerv > > ___ > dev-security-policy mailing list > dev-security-policy@lists.mozilla.org > https://lists.mozilla.org/listinfo/dev-security-policy > > ___ dev-security-policy mailing list dev-security-policy@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-security-policy
RE: FW: StartCom inclusion request: next steps
> On 15/09/17 11:01, Inigo Barreira wrote: > > Considering that we were distrusted, that we didn´t reapply for > > inclussion, that CT is only required by Chrome and it´s not included > > in the Mozilla policy (even we were requested that all of our certs > > had to be CT logged) nor required by Firefox, that those certs were > > under our control all the time and lived for some minutes because were > > revoked inmediately, at that time, when we did it, we didn´t expect > > this reaction for sure. > > But surely CT testing is not the only sort of testing you've been doing? Yes, this is the only test we did it in production > E.g. you made some test certificates with different types of ECC curve, which > you then had to revoke some of as against browser policies. No, those weren´t tests. We allowed the use of curves permitted by the BRs but this issue came up in the mozilla policy (I think Arkadiusz posted) and I also asked about it in the last CABF F2F (I asked Ryan about it) and then, with that outcome and as the browsers didn´t accept them, we revoked and then not allow the issuance. I think the discussion is still active (i.e. the use of P-521). > If these had been in a testing hierarchy there would have been no problem. > > CAs have been heavily criticised over the past few years for issuing test > certificates in public hierarchies (see e.g. Symantec). The danger of doing so > should be well known to all CAs by now. Yes, I know. But the only testing we did in production was the one related to the CT. > > Perhaps once a test has been passed and checked in a testing system, and if > the certificates concerned do not violate any policies, it could be repeated > on > a production system to deal with any possible differences between the two. > But starting with the production system is not a good idea. True, but it seems you´re understanding that we have only a production system in which we test everything and this is not the case. Before moving anything into production, we have tested in development and in the QA system. > > Gerv smime.p7s Description: S/MIME cryptographic signature ___ dev-security-policy mailing list dev-security-policy@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-security-policy
Re: StartCom inclusion request: next steps
On 15/09/17 08:53, Inigo Barreira wrote: > Yes, you´re right, that was on the table and also suggested by Mozilla, but > the issue was that people from 360 are used to code in PHP and the old one > was in Java and some other for which they are not so familiar and then was > decided to re-write all the code in PHP trying to keep the same > functionality. Given the quality of code produced, it might have been better in hindsight to hire Java experts to work on the old codebase. > Furthermore, with this decission, we also wanted to let the community know > that this is totally a new CA system in all aspects, nothing related to the > past, everything from scratch, so new coding, new programming language, new > PKI system, infrastructure, etc. hoping this would make the community have a > better impression of the new Startcom regarding the distrust issue. "We rewrote everything from scratch" is not actually something which itself inspires confidence. In the case of WoSign, it was required of them because their old code was clearly terrible and buggy. But the reason the result would have to be strongly audited (were they to reapply) is that new code is riskier than old, tried-and-tested code. I don't know if I ever wrote it down anywhere, but I'm fairly sure that switching back from the WoSign codebase to the older StartCom codebase (i.e. reversing the change you made after the purchase) was my suggestion for how StartCom should proceed after the dis-trust event. That doesn't mean you are required to take my advice, but it might have been a hint that I wouldn't consider "hey, we rewrote everything from scratch!" as a positive point. Gerv ___ dev-security-policy mailing list dev-security-policy@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-security-policy
Re: FW: StartCom inclusion request: next steps
On 15/09/17 11:01, Inigo Barreira wrote: > Considering that we were distrusted, that we didn´t reapply for > inclussion, that CT is only required by Chrome and it´s not included > in the Mozilla policy (even we were requested that all of our certs > had to be CT logged) nor required by Firefox, that those certs were > under our control all the time and lived for some minutes because > were revoked inmediately, at that time, when we did it, we didn´t > expect this reaction for sure. But surely CT testing is not the only sort of testing you've been doing? E.g. you made some test certificates with different types of ECC curve, which you then had to revoke some of as against browser policies. If these had been in a testing hierarchy there would have been no problem. CAs have been heavily criticised over the past few years for issuing test certificates in public hierarchies (see e.g. Symantec). The danger of doing so should be well known to all CAs by now. Perhaps once a test has been passed and checked in a testing system, and if the certificates concerned do not violate any policies, it could be repeated on a production system to deal with any possible differences between the two. But starting with the production system is not a good idea. Gerv ___ dev-security-policy mailing list dev-security-policy@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-security-policy
Re: FW: StartCom inclusion request: next steps
On 15/09/17 09:24, Inigo Barreira wrote: > AFAIK, Certinomis only disclosed in the CCADB That means it's published and available. As noted in my other reply, information as to exactly what this cross-sign enables trust for would be most helpful, as I may have misunderstood previous statements on this topic. Gerv ___ dev-security-policy mailing list dev-security-policy@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-security-policy
Re: FW: StartCom inclusion request: next steps
Hi Inigo, On 14/09/17 16:05, Inigo Barreira wrote: > Those tests were done to check the CT behaviour, there was any other testing > of the new systems, just for the CT. Is there any reason those tests could not have been done using a parallel testing hierarchy (other than the fact that you hadn't set one up)? > Some of these "mis-issuances" were due to some incongruencies between the BRs > and the Mozilla policy, such as the use of different curves (allowed by the > BRs but not for some browsers), But it is the job of a CA to be aware of browser policies. > As far as I know, the current manager of Startcom has not been previously > accused of deception or bad action. No, indeed. There is no accusation that you have been deceptive towards anyone. > Yes, it´s not a policy violation. As explained, this was a problem in the > EJBCA with the UTF8 encoding. That was the reason for the revocation; it's not the reason for the key reuse. >> * The WT/BR/EV audits on StartCom's website are significantly qualified, and >> they include lack of controls on issuance. They should have clean ones done >> before we permit any inclusion request to proceed. The qualifications >> include: >> >> - Risk analysis process defined but not implemented >> - Business continuity plan defined but not implemented >> - Audit logs not guaranteed to have integrity >> - Monitoring system cannot detect security-related changes to >> Certificate Systems > > Yes, this is also true. Our webtrust audits have findings but those are not > so significant according to the auditors who signed the reports, so I assume > the auditors thought that the system is good enough to have the audit report > in place. I think lack of monitoring and lack of integrity of logs are serious issues. Repairing them afterwards does not remove the uncertainty. If you said "I left the root certificate private key on a USB stick on the desk in my unlocked office over the weekend - but it's OK, I've remediated the problem now, and it's back in the safe", that would not remove the uncertainty about whether someone had done something with it in the mean time. > I don´t know how these you mention have applied, but I remember lots of > issues regarding Google and the acquisition of the Globalsign roots and how > they proceeded. I think "lots of issues" is an overstatement. There was an issue regarding the scope of their audits which was raised publicly, but it was something that Google has explicitly sought our approval for in advance. > Not sure about this. We were distrusted in october 2016, the new system > started to operate in april 2017, which is not related to the old one which > has been switched off. None of the new certs are trusted in Mozilla Firefox > and we notify our users so by messages in the web and applications. I may have made a mistake here. I was under the impression that you had told me that your new hierarchy, cross-signed by Certnomis, had issued 50,000 certificates. Did I misunderstand? If so, my apologies. > Ok, I see this is a new requirement that was not imposed last time in which > you recommended and allowed us to be cross-signed as many other CAs have done > in the past to be in the business. Mozilla did not recommend that you be cross-signed. Certnomis raised the possibility, and we said that it could happen when you had met the BRs and completed the remediation steps. The plan we "approved" was not a cross-signing plan. The audits show you haven't yet met the BRs, and we have not yet signed off on you having completed the remediation steps. > We´ve met all the conditions, new system, new management, security audit and > webtrust audit and CT logging. In those conditions, it was not mentioned that > the webtrust audit should be clean Isn't that implied? > I´ve never said this. In fact, despite having that cross-signed which were > provided to us in july we have never used and provided to any of our > customers to build a trusted path. So none of those 5, or the new ones, > go with the Certinomis path because none have it. But all those 5 certs > are untrusted because we´re not in the Mozilla root, not the new one, and the > old one was distrusted. So the 50,000 certs you mentioned are issued in your old hierarchy, which is not cross-signed by Certnomis? If you could clarify the numbers for your old and new PKI, and confirm that the Certnomis cross-signs apply only to the new one, that would be helpful. Again, apologies for any misunderstanding on my part. > In fact, recently, I asked for permission to use the Certinomis cross-signed > certificates and have no response. I don´t know if this is an administrative > silence which may allow me to use it but until having a clear direction we > haven´t used it. Can you remind me how you asked and when? > I think this has been explained. I don´t understand why you say it´s a "poor > quality PHP code". Because poor quality PHP code does n
RE: FW: StartCom inclusion request: next steps
> Hi Inigo, > > To add from the last post. > > I know this is unwelcome news to you but I feel that with all these incidents > happening right now with Symantec and the incidents before, we can't really > take any more chances. Every incident is eroding trust in this system and if > we > want more people to take up adoption of https in the long term, I feel that > we need to start operating infrastructure above board and without issues. > > James James, I share with you this desire. But we´re seeing one day and another that issues are still happening, and not all are from StartCom :-) so to avoid eroding trust what do you suggest if not give more chances? Remove them all? > ___ > dev-security-policy mailing list > dev-security-policy@lists.mozilla.org > https://lists.mozilla.org/listinfo/dev-security-policy smime.p7s Description: S/MIME cryptographic signature ___ dev-security-policy mailing list dev-security-policy@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-security-policy
RE: FW: StartCom inclusion request: next steps
> > > > > > > Those tests were done to check the CT behaviour, there was any > > > > other > > > testing of the new systems, just for the CT. Those certs were under > > > control all the time and were lived for some minutes because were > > > revoked inmediately after checking the certs were logged correctly > > > in the CTs. It´s not a mis- issuance by means of we didn´t know what > > > happened, we had to investigate, etc. It was not a good practice and > > > I can´t excuse for that, but it was not related to the regular > > > issuance procedure as someone suggested. We provided a report in > > > which indicated all that happened and what we did to not happen this > again, updating the EJBCA roles permissions. > > > > > > 1) Why didn't StartCom build a test hierarchy? > > > > Considering that we were distrusted, that we didn´t reapply for inclussion, > that CT is only required by Chrome and it´s not included in the Mozilla policy > (even we were requested that all of our certs had to be CT logged) nor > required by Firefox, that those certs were under our control all the time and > lived for some minutes because were revoked inmediately, at that time, when > we did it, we didn´t expect this reaction for sure. > > > > Of course if we had known it we hadn´t done it and for sure had built > > a test hierarchy but there´s nothing we can do now. Only wanted to > > state that those certs were under our control all the time, and lived > > for some minutes because were revoked after the test. There were not > > any other testing of any other nature directly in the production > > system > > > > > 2) Why didn't StartCom use the TestTube CT log for testing CT? > > > > We tried to check and test the same behaviour before going live with the CT > logging, so followed the requirements to use 3 logs, one google and one non- > google, for the EVs and this is what we did. We used the same settings we > had before the distrust using the startcom log and the google ones (pilot and > rocketeer). > > Hi Inigo, > > I'm just trying to get everything straight, cleared up and then we can move > on. > > I found all of your answers very concerning in the way you've conducted > testing. I thought that all CAs must have some type of test hierarchy in place > to test new software, Well, this is not required. Of course we have a development and a testing system, which are not "connected" into production. As we wanted to check the same behaviour for the CT ecosystem than before the distrust, we did it directly in production, but don´t take this as we didn´t test it before in our development and testing systems, for sure we did. > requirements and etc before going live but from the answers you've given, > StartCom neglected this in favour to test as you go > along and deal with the problems using a live CA hierarchy. As said, this wasn´t a "live" CA hierarchy. We were distrusted, nothing issued from that "live" CA was trusted, this wasn´t any harm to any of the Mozilla users. > All this feels very amateurish and doesn't give me the any confidence at all > that your CA is > proven itself to be a trustworthy part of this infrastructure. I respect this opinion but can´t agree with it. > > I recommend to Mozilla to require StartCom to start again fresh. > > 1. Build a test hierarchy. Test the software and etc to industry guidelines. > 2. Build a new production hierarchy (including new HSMs, keys, roots, etc.) > and then re-apply for inclusion into the Mozilla root program. > 3. Once approved then you can cross-sign your roots with another CA. > > While this is happening. You can resell certificates from other CAs and build > up trust in the industry which will benefit you in the long term I feel. > > James > ___ > dev-security-policy mailing list > dev-security-policy@lists.mozilla.org > https://lists.mozilla.org/listinfo/dev-security-policy smime.p7s Description: S/MIME cryptographic signature ___ dev-security-policy mailing list dev-security-policy@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-security-policy
Re: FW: StartCom inclusion request: next steps
On Friday, September 15, 2017 at 12:30:00 PM UTC+1, James Burton wrote: > On Friday, September 15, 2017 at 10:56:11 AM UTC+1, Inigo Barreira wrote: > > > > > > > Those tests were done to check the CT behaviour, there was any other > > > testing of the new systems, just for the CT. Those certs were under > > > control all > > > the time and were lived for some minutes because were revoked inmediately > > > after checking the certs were logged correctly in the CTs. It´s not a mis- > > > issuance by means of we didn´t know what happened, we had to investigate, > > > etc. It was not a good practice and I can´t excuse for that, but it was > > > not > > > related to the regular issuance procedure as someone suggested. We > > > provided a report in which indicated all that happened and what we did to > > > not happen this again, updating the EJBCA roles permissions. > > > > > > 1) Why didn't StartCom build a test hierarchy? > > > > Considering that we were distrusted, that we didn´t reapply for inclussion, > > that CT is only required by Chrome and it´s not included in the Mozilla > > policy (even we were requested that all of our certs had to be CT logged) > > nor required by Firefox, that those certs were under our control all the > > time and lived for some minutes because were revoked inmediately, at that > > time, when we did it, we didn´t expect this reaction for sure. > > > > Of course if we had known it we hadn´t done it and for sure had built a > > test hierarchy but there´s nothing we can do now. Only wanted to state that > > those certs were under our control all the time, and lived for some minutes > > because were revoked after the test. There were not any other testing of > > any other nature directly in the production system > > > > > 2) Why didn't StartCom use the TestTube CT log for testing CT? > > > > We tried to check and test the same behaviour before going live with the CT > > logging, so followed the requirements to use 3 logs, one google and one > > non-google, for the EVs and this is what we did. We used the same settings > > we had before the distrust using the startcom log and the google ones > > (pilot and rocketeer). > > Hi Inigo, > > I'm just trying to get everything straight, cleared up and then we can move > on. > > I found all of your answers very concerning in the way you've conducted > testing. I thought that all CAs must have some type of test hierarchy in > place to test new software, requirements and etc before going live but from > the answers you've given, StartCom neglected this in favour to test as you go > along and deal with the problems using a live CA hierarchy. All this feels > very amateurish and doesn't give me the any confidence at all that your CA is > proven itself to be a trustworthy part of this infrastructure. > > I recommend to Mozilla to require StartCom to start again fresh. > > 1. Build a test hierarchy. Test the software and etc to industry guidelines. > 2. Build a new production hierarchy (including new HSMs, keys, roots, etc.) > and then re-apply for inclusion into the Mozilla root program. > 3. Once approved then you can cross-sign your roots with another CA. > > While this is happening. You can resell certificates from other CAs and build > up trust in the industry which will benefit you in the long term I feel. > > James Hi Inigo, To add from the last post. I know this is unwelcome news to you but I feel that with all these incidents happening right now with Symantec and the incidents before, we can't really take any more chances. Every incident is eroding trust in this system and if we want more people to take up adoption of https in the long term, I feel that we need to start operating infrastructure above board and without issues. James ___ dev-security-policy mailing list dev-security-policy@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-security-policy
Re: FW: StartCom inclusion request: next steps
On Friday, September 15, 2017 at 10:56:11 AM UTC+1, Inigo Barreira wrote: > > > > > Those tests were done to check the CT behaviour, there was any other > > testing of the new systems, just for the CT. Those certs were under control > > all > > the time and were lived for some minutes because were revoked inmediately > > after checking the certs were logged correctly in the CTs. It´s not a mis- > > issuance by means of we didn´t know what happened, we had to investigate, > > etc. It was not a good practice and I can´t excuse for that, but it was not > > related to the regular issuance procedure as someone suggested. We > > provided a report in which indicated all that happened and what we did to > > not happen this again, updating the EJBCA roles permissions. > > > > 1) Why didn't StartCom build a test hierarchy? > > Considering that we were distrusted, that we didn´t reapply for inclussion, > that CT is only required by Chrome and it´s not included in the Mozilla > policy (even we were requested that all of our certs had to be CT logged) nor > required by Firefox, that those certs were under our control all the time and > lived for some minutes because were revoked inmediately, at that time, when > we did it, we didn´t expect this reaction for sure. > > Of course if we had known it we hadn´t done it and for sure had built a test > hierarchy but there´s nothing we can do now. Only wanted to state that those > certs were under our control all the time, and lived for some minutes because > were revoked after the test. There were not any other testing of any other > nature directly in the production system > > > 2) Why didn't StartCom use the TestTube CT log for testing CT? > > We tried to check and test the same behaviour before going live with the CT > logging, so followed the requirements to use 3 logs, one google and one > non-google, for the EVs and this is what we did. We used the same settings we > had before the distrust using the startcom log and the google ones (pilot and > rocketeer). Hi Inigo, I'm just trying to get everything straight, cleared up and then we can move on. I found all of your answers very concerning in the way you've conducted testing. I thought that all CAs must have some type of test hierarchy in place to test new software, requirements and etc before going live but from the answers you've given, StartCom neglected this in favour to test as you go along and deal with the problems using a live CA hierarchy. All this feels very amateurish and doesn't give me the any confidence at all that your CA is proven itself to be a trustworthy part of this infrastructure. I recommend to Mozilla to require StartCom to start again fresh. 1. Build a test hierarchy. Test the software and etc to industry guidelines. 2. Build a new production hierarchy (including new HSMs, keys, roots, etc.) and then re-apply for inclusion into the Mozilla root program. 3. Once approved then you can cross-sign your roots with another CA. While this is happening. You can resell certificates from other CAs and build up trust in the industry which will benefit you in the long term I feel. James ___ dev-security-policy mailing list dev-security-policy@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-security-policy
RE: FW: StartCom inclusion request: next steps
> > > Those tests were done to check the CT behaviour, there was any other > testing of the new systems, just for the CT. Those certs were under control > all > the time and were lived for some minutes because were revoked inmediately > after checking the certs were logged correctly in the CTs. It´s not a mis- > issuance by means of we didn´t know what happened, we had to investigate, > etc. It was not a good practice and I can´t excuse for that, but it was not > related to the regular issuance procedure as someone suggested. We > provided a report in which indicated all that happened and what we did to > not happen this again, updating the EJBCA roles permissions. > > 1) Why didn't StartCom build a test hierarchy? Considering that we were distrusted, that we didn´t reapply for inclussion, that CT is only required by Chrome and it´s not included in the Mozilla policy (even we were requested that all of our certs had to be CT logged) nor required by Firefox, that those certs were under our control all the time and lived for some minutes because were revoked inmediately, at that time, when we did it, we didn´t expect this reaction for sure. Of course if we had known it we hadn´t done it and for sure had built a test hierarchy but there´s nothing we can do now. Only wanted to state that those certs were under our control all the time, and lived for some minutes because were revoked after the test. There were not any other testing of any other nature directly in the production system > 2) Why didn't StartCom use the TestTube CT log for testing CT? We tried to check and test the same behaviour before going live with the CT logging, so followed the requirements to use 3 logs, one google and one non-google, for the EVs and this is what we did. We used the same settings we had before the distrust using the startcom log and the google ones (pilot and rocketeer). smime.p7s Description: S/MIME cryptographic signature ___ dev-security-policy mailing list dev-security-policy@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-security-policy
Re: FW: StartCom inclusion request: next steps
> Those tests were done to check the CT behaviour, there was any other testing > of the new systems, just for the CT. Those certs were under control all the > time and were lived for some minutes because were revoked inmediately after > checking the certs were logged correctly in the CTs. It´s not a mis-issuance > by means of we didn´t know what happened, we had to investigate, etc. It was > not a good practice and I can´t excuse for that, but it was not related to > the regular issuance procedure as someone suggested. We provided a report in > which indicated all that happened and what we did to not happen this again, > updating the EJBCA roles permissions. 1) Why didn't StartCom build a test hierarchy? 2) Why didn't StartCom use the TestTube CT log for testing CT? ___ dev-security-policy mailing list dev-security-policy@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-security-policy
RE: FW: StartCom inclusion request: next steps
> On 14/09/2017 17:05, Inigo Barreira wrote: > > All, > > > > ... > >> > >> We should add the existing Certnomis cross-signs to OneCRL to revoke > >> all the existing certificates. As of 10th August (now a month ago) > >> StartCom said they have 5 outstanding SSL certs which are valid > >> due to the Certnomis cross- sign. > > > > I´ve never said this. In fact, despite having that cross-signed which were > provided to us in july we have never used and provided to any of our > customers to build a trusted path. So none of those 5, or the new ones, > go with the Certinomis path because none have it. But all those 5 certs > are untrusted because we´re not in the Mozilla root, not the new one, and the > old one was distrusted. > > In fact, recently, I asked for permission to use the Certinomis cross-signed > certificates and have no response. I don´t know if this is an administrative > silence which may allow me to use it but until having a clear direction we > haven´t used it. > > > > > I can't speak for Mozilla, but the obvious point is, that as soon as the cross > certificate from Certinomis was published in CT-logs or elsewhere, any > StartCom customer could (and still can) download it and install it on their > server, thus activating the Certinomis path for their server. > AFAIK, Certinomis only disclosed in the CCADB and even we received it, we have not sent to any customer nor posted anywhere. And I know crt.sh includes it and when questioned for it Rob answered that crt.sh includes all certs that have been submitted to one or more of the monitored CT logs, and includes all trust paths that can be built. I don´t think any StartCom customer has downloaded from wherever it could be and installed and created the trusted path. Our customers base are not very familiar with this stuff. Usually we need to provide all of them with instructions on what and how to do it. > > Enjoy > > Jakob > -- > Jakob Bohm, CIO, Partner, WiseMo A/S. https://www.wisemo.com > Transformervej 29, 2860 Søborg, Denmark. Direct +45 31 13 16 10 This > public discussion message is non-binding and may contain errors. > WiseMo - Remote Service Management for PCs, Phones and Embedded > ___ > dev-security-policy mailing list > dev-security-policy@lists.mozilla.org > https://lists.mozilla.org/listinfo/dev-security-policy smime.p7s Description: S/MIME cryptographic signature ___ dev-security-policy mailing list dev-security-policy@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-security-policy
RE: FW: StartCom inclusion request: next steps
Yes, there are similar ones everywhere, so I´m familiar with it :-) And you´re right, I also make contributions in many other places, ETSI, ENISA, CABF (used to), ... and not get paid for that, but it´s also true that the way the distrust happened didn´t give us time or much time to act accordingly, even now people is asking for the distrust or asking for refunds due to the distrust. And yes, we tried to do it quickly and we know what happened but we didn´t re-apply at that time. We got issues and until all of them were fixed and we receive the OK we didn´t move forward. We wanted to meet the deadline we imposed ourselves and even reached, it was not in a good shape, so, had to start again. And that´s what I said internally, that it didn´t matter if we could not make it in 6 months as indicated in the sanction and it took us nearly 10, taking into account that everything was new. Best regards Iñigo Barreira CEO StartCom CA Limited > -Original Message- > From: dev-security-policy [mailto:dev-security-policy- > bounces+inigo=startcomca@lists.mozilla.org] On Behalf Of Nick Lamb via > dev-security-policy > Sent: viernes, 15 de septiembre de 2017 1:22 > To: mozilla-dev-security-pol...@lists.mozilla.org > Subject: Re: FW: StartCom inclusion request: next steps > > On Thursday, 14 September 2017 16:00:35 UTC+1, Inigo Barreira wrote: > > Well, finally this is a business and I don´t think none on this list is > > working > for free. At the end everyone has his/her salary, etc. But that was not the > main reason because getting included in the root programs takes time but > wanted to provide our customers which gave us support for what happened > with the distrust (which IMHO in the case of Startcom was very aggressive) a > solution generating a new fresh and clean system. > > I can't speak for other people contributing to m.d.s.policy but of course I am > not paid to do this. I have a job, but it isn't this. I have never asked my > employer's permission to contribute here, judging it to be entirely outside > their purview. My job does include running a (small, private) CA, but then it > also includes maintaining a DBMS, a web crawler and all sorts of other stuff, > I'm sure it would be possible to identify some connection to my job for almost > any technical contribution I could make anywhere. > > There is an proverb in English, I am not sure if you're familiar with it, > "More > haste, less speed". What this means is that in trying to perform a task as > quickly as possible you may instead cause yourself such extra trouble that in > the end the task takes even longer to complete. I am sure that even if you > have not come across a phrase like this before you can see its applicability > to > the situation for StartCom. > ___ > dev-security-policy mailing list > dev-security-policy@lists.mozilla.org > https://lists.mozilla.org/listinfo/dev-security-policy smime.p7s Description: S/MIME cryptographic signature ___ dev-security-policy mailing list dev-security-policy@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-security-policy
RE: StartCom inclusion request: next steps
Hi Percy, Yes, you´re right, that was on the table and also suggested by Mozilla, but the issue was that people from 360 are used to code in PHP and the old one was in Java and some other for which they are not so familiar and then was decided to re-write all the code in PHP trying to keep the same functionality. Besides, the old code had to be integrated with the new PKI infrastructure, EJBCA, and that was not an easy task. All in all on the table, integration with new systems, maintenance of the old code if issues arise due to this integration or any other problem, not good language knowledge, get used to PHP, etc. made us took that decission. Furthermore, with this decission, we also wanted to let the community know that this is totally a new CA system in all aspects, nothing related to the past, everything from scratch, so new coding, new programming language, new PKI system, infrastructure, etc. hoping this would make the community have a better impression of the new Startcom regarding the distrust issue. Best regards Iñigo Barreira CEO StartCom CA Limited > -Original Message- > From: dev-security-policy [mailto:dev-security-policy- > bounces+inigo=startcomca@lists.mozilla.org] On Behalf Of Percy via dev- > security-policy > Sent: jueves, 14 de septiembre de 2017 22:13 > To: mozilla-dev-security-pol...@lists.mozilla.org > Subject: Re: StartCom inclusion request: next steps > > "Conclusion: StartCom's attempt to restart the CA was rushed." > > "It was a very hard task in very few time but the people at 360 tried > everything to get it done by that date, end of december 2016, and yes, we > reached the date but with many failures" > > May I ask why StartCom choose to rush everything in PHP from the ground up > rather than using the more secure system already in place in the old > StartCom? From my understanding, the distrust of StartCom is more related > to the secret acquisition by WoSign an Qihoo 360 rather than insecure > infrastructure. So if the deadline is so imminent as you stated and pressure is > so high from customers, can't you use the reasonably secure old code base > rather than rushing everything from the ground up? Then you will have more > time transition to another system if needed with sufficient time for secure > processes? > ___ > dev-security-policy mailing list > dev-security-policy@lists.mozilla.org > https://lists.mozilla.org/listinfo/dev-security-policy smime.p7s Description: S/MIME cryptographic signature ___ dev-security-policy mailing list dev-security-policy@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-security-policy
Re: FW: StartCom inclusion request: next steps
On 14/09/2017 17:05, Inigo Barreira wrote: All, ... We should add the existing Certnomis cross-signs to OneCRL to revoke all the existing certificates. As of 10th August (now a month ago) StartCom said they have 5 outstanding SSL certs which are valid due to the Certnomis cross- sign. I´ve never said this. In fact, despite having that cross-signed which were provided to us in july we have never used and provided to any of our customers to build a trusted path. So none of those 5, or the new ones, go with the Certinomis path because none have it. But all those 5 certs are untrusted because we´re not in the Mozilla root, not the new one, and the old one was distrusted. In fact, recently, I asked for permission to use the Certinomis cross-signed certificates and have no response. I don´t know if this is an administrative silence which may allow me to use it but until having a clear direction we haven´t used it. I can't speak for Mozilla, but the obvious point is, that as soon as the cross certificate from Certinomis was published in CT-logs or elsewhere, any StartCom customer could (and still can) download it and install it on their server, thus activating the Certinomis path for their server. Enjoy Jakob -- Jakob Bohm, CIO, Partner, WiseMo A/S. https://www.wisemo.com Transformervej 29, 2860 Søborg, Denmark. Direct +45 31 13 16 10 This public discussion message is non-binding and may contain errors. WiseMo - Remote Service Management for PCs, Phones and Embedded ___ dev-security-policy mailing list dev-security-policy@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-security-policy
Re: FW: StartCom inclusion request: next steps
On Thursday, 14 September 2017 16:00:35 UTC+1, Inigo Barreira wrote: > Well, finally this is a business and I don´t think none on this list is > working for free. At the end everyone has his/her salary, etc. But that was > not the main reason because getting included in the root programs takes time > but wanted to provide our customers which gave us support for what happened > with the distrust (which IMHO in the case of Startcom was very aggressive) a > solution generating a new fresh and clean system. I can't speak for other people contributing to m.d.s.policy but of course I am not paid to do this. I have a job, but it isn't this. I have never asked my employer's permission to contribute here, judging it to be entirely outside their purview. My job does include running a (small, private) CA, but then it also includes maintaining a DBMS, a web crawler and all sorts of other stuff, I'm sure it would be possible to identify some connection to my job for almost any technical contribution I could make anywhere. There is an proverb in English, I am not sure if you're familiar with it, "More haste, less speed". What this means is that in trying to perform a task as quickly as possible you may instead cause yourself such extra trouble that in the end the task takes even longer to complete. I am sure that even if you have not come across a phrase like this before you can see its applicability to the situation for StartCom. ___ dev-security-policy mailing list dev-security-policy@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-security-policy
Re: StartCom inclusion request: next steps
"Conclusion: StartCom's attempt to restart the CA was rushed." "It was a very hard task in very few time but the people at 360 tried everything to get it done by that date, end of december 2016, and yes, we reached the date but with many failures" May I ask why StartCom choose to rush everything in PHP from the ground up rather than using the more secure system already in place in the old StartCom? From my understanding, the distrust of StartCom is more related to the secret acquisition by WoSign an Qihoo 360 rather than insecure infrastructure. So if the deadline is so imminent as you stated and pressure is so high from customers, can't you use the reasonably secure old code base rather than rushing everything from the ground up? Then you will have more time transition to another system if needed with sufficient time for secure processes? ___ dev-security-policy mailing list dev-security-policy@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-security-policy
FW: StartCom inclusion request: next steps
All, Obviously this is not the message we would like to read and will try to explain and rebate as much as possible some of the comments posted here. > > The Mozilla CA Certificates team has been considering what the appropriate > next steps are for the inclusion request from the CA "StartCom".[0] As readers > will know, this CA has previously been removed from trust[1], and so a re- > application obviously involves particular scrutiny. In addition, several > questions have been raised about the way in which the new StartCom PKI has > been operated technically[2]. This is a proposal for the way forward, on which > comments are invited. > > Mozilla's considered view is as follows: > > * It should have been obvious to StartCom that testing of their new systems > needed to be done using a parallel testing hierarchy. Those tests were done to check the CT behaviour, there was any other testing of the new systems, just for the CT. Those certs were under control all the time and were lived for some minutes because were revoked inmediately after checking the certs were logged correctly in the CTs. It´s not a mis-issuance by means of we didn´t know what happened, we had to investigate, etc. It was not a good practice and I can´t excuse for that, but it was not related to the regular issuance procedure as someone suggested. We provided a report in which indicated all that happened and what we did to not happen this again, updating the EJBCA roles permissions. > That it was not obvious, is deeply concerning. It is also concerning that > someone can sit at a terminal > and issue random certificates with variable values in lots of fields, in what > is > to become a publicly-trusted hierarchy. Well, it was possible at that time, but only the CA administrator could do it and under many requirements. It´s not like sitting at a terminal and start issuing certificates, there were and are security mechanisms to avoid "someone" could do that and I can list many. Probably most of the CA administrators of the rest of CAs had this capacity (maybe not now) because the majority of the PKI software allows it and it´s needed when building a hierarchy. > It's not about numbers (e.g. "40 out of 5"), it's about the process. > This number of 40 is about the total of "mis-issuances" discovered, not only related to these ones for the CT testing. And some other times, discussed in this list, the number matters. Even more, for those 40, most of them were "discovered" by us and acted accordingly as per the BRs. We revoked the majority of them within the 24 hours of being notified internally. When those were posted in the bugzilla, as said, most were revoked and started the investigation on what happened and what actions needed to be done. Some of these "mis-issuances" were due to some incongruencies between the BRs and the Mozilla policy, such as the use of different curves (allowed by the BRs but not for some browsers), or about pre-certificates in which is not clear if they fall under what requirement as a discussion started by Jeremy on the list. For example, is it necessary to revoke also the pre-certificates when a certificate is revoked? Are they need to be considered certificates and meet the BRs and/or Mozilla policy? Or about the use of Unicode vs punnycode which is still under discussions, even a ballot failed in the CABF. So, those errors we made were also made by some others, and not being as an excuse, but it seems that it was not clear for the CAs. We updated our procedure issuance to avoid these issues happening in mid July. What did we do? - Restrict the use of eliptic curves only to those admitted by Mozilla - Change the certificate profile for not having differences with the key encipherment and key agreement - update the internal db for country codes - update the sytems for changing all domain names to punnycode - and recently develop a csr checking tool to avoid the issue with RSA parameters because EJBCA didn´t have it at that time (it comes now with the new release 6.9.0) > As JC Jones wrote: > > "This is a professional PKI operation, being overseen by industry veterans. If > something as concrete as the issuance process had such a glaring quality > assurance methodology failure, why should anyone believe that something > much harder -- subscriber validation -- is going to be done correctly?" Well, this is an opinion. And I fully respect but none is free of failures and let´s encrypt (and many others) is also having them as we´re seeing recently with weak keys, etc. and I´m none to say they are not professional, or not having a quality assurance methodology, ... or for not believing they are not acting correctly. For sure they are, but same as us. I can´t critize anyone and not because we´re in a weak position at Startcom in which everyone is looking deeply what we do, scrutinizing deeply. For all these failures we acted quickly because our internal proc
Re: StartCom inclusion request: next steps
> On Sep 14, 2017, at 04:49, Gervase Markham via dev-security-policy > wrote: > > We should add the existing Certnomis cross-signs to OneCRL to revoke > all the existing certificates. As of 10th August (now a month ago) > StartCom said they have 5 outstanding SSL certs which are valid due > to the Certnomis cross-sign. Revoking them all by adding intermediates > to OneCRL would therefore lead to non-negligible disruption. But these > were issued by an org whose most recent audits are qualified, which is > under sanction, and about whose issuance practices and process safety > there is a reasonable amount of doubt. We may allow a grace period for > customers to replace them with certs from a trusted provider. I’m not yet convinced a grace period is necessary. StartCom does not list Firefox as a compatible browser on their website (they have the logos for Internet Explorer, Microsoft Edge, Android, and Windows). Additionally, there are multiple steps in the StartCom issuance flow that contain the following in red text: > Notice: > 1. Mozilla and Google decided to distrust all StartCom root certificates as > of 21st of October, 2016, meaning that since January all the SSL certificates > issued from that date will no longer be trusted in Firefox and Chrome newest > releases. > Besides, Google has gone further and as explained here: > https://security.googleblog.com/2016/10/distrusting-wosign-and-startcom.html > will not trust even those SSL certificates issued before that date until the > final disruption. > Apple's decision announced on Nov 30th, 2016 was to distrust all StartCom > root certificates as of 1st of December, meaning that SSL cert issued after > December 1st, 2016 will no longer be trusted in Apple’s systems. > 2. Any subscribers that paid the validation fee after Oct. 21st, 2016 can get > full refund by request. > 3. Currently StartCom is able to provide an interim solution for organization > users in case of requested. > Meanwhile StartCom is updating all systems and is following all requirements > requested by Mozilla to regain the trust in these browsers and re-apply after > the 6 months time penalty. Given this explicit notification, I do not believe that subscribers would ever have had the expectation that their certificates would work with Firefox. Jonathan ___ dev-security-policy mailing list dev-security-policy@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-security-policy
StartCom inclusion request: next steps
The Mozilla CA Certificates team has been considering what the appropriate next steps are for the inclusion request from the CA "StartCom".[0] As readers will know, this CA has previously been removed from trust[1], and so a re-application obviously involves particular scrutiny. In addition, several questions have been raised about the way in which the new StartCom PKI has been operated technically[2]. This is a proposal for the way forward, on which comments are invited. Mozilla's considered view is as follows: * It should have been obvious to StartCom that testing of their new systems needed to be done using a parallel testing hierarchy. That it was not obvious, is deeply concerning. It is also concerning that someone can sit at a terminal and issue random certificates with variable values in lots of fields, in what is to become a publicly-trusted hierarchy. It's not about numbers (e.g. "40 out of 5"), it's about the process. As JC Jones wrote: "This is a professional PKI operation, being overseen by industry veterans. If something as concrete as the issuance process had such a glaring quality assurance methodology failure, why should anyone believe that something much harder -- subscriber validation -- is going to be done correctly?" * The key for their new root certificate was also used in a couple of intermediates (one revoked as it was done incorrectly - again, lack of testing!). While this is probably not a policy violation, it's not good practice. * StartCom's infrastructure audit, performed by Cure53, was frankly a security disaster. (They are using EJBCA for CA operations; this was an audit of their front end and customer management systems, which were rewritten by a team from their new owner, Qihoo 360.) The (PHP) codebase was full of holes, poorly commented, had few or no tests, and showed every evidence of being hacked together in an enormous rush. This does not inspire confidence. Cure53 say they retested a couple of months later and most of the holes they found were fixed - although they found quite a few more. All this does not bode well - Cure53 are not infallible, an audit is not a substitute for secure coding practices, and the initial results show that the software was clearly not built by people who understand software security. The summary of their results is attached to their Action Items bug[3], but it does leave out some of the more critical passages of commentary from the original, and of course does not show the particular holes found and their scope and severity. * The WT/BR/EV audits on StartCom's website are significantly qualified, and they include lack of controls on issuance. They should have clean ones done before we permit any inclusion request to proceed. The qualifications include: - Risk analysis process defined but not implemented - Business continuity plan defined but not implemented - Audit logs not guaranteed to have integrity - Monitoring system cannot detect security-related changes to Certificate Systems * Certnomis chose to cross-sign StartCom while StartCom had audits with significant qualifications, and allowed them to recommence publicly-trusted issuance before they had demonstrated to Mozilla that they had met the remediation conditions required. While this may not have been against the letter of our requirements for StartCom to restart trusted operations, we feel it was not in the spirit of them. All in all, this attempt to start a new CA compares poorly with other recent executions of this process, such as those by Google and Amazon. While those companies do have significantly more resources than StartCom, many of the issues raised are questions of good practice, not of money. Conclusion: StartCom's attempt to restart the CA was rushed. One could speculate why that was; perhaps due to a requirement to start generating income again. But a process of building a production PKI by trial and error, revoking your mistakes, is entirely inappropriate. The qualified audits include missing or unimplemented processes, and audit/monitoring failures which lead to uncertainty as to how well the new roots were protected. This all shows that StartCom were not ready to start up the production PKI when they did. And yet Firefox today trusts tens of thousands of certificates issued by this PKI. Considering all this, our proposal is to require that StartCom begin again with new-new roots. These roots should be generated inside an already-security-validated infrastructure, as part of a new WT/BR audit process, the end results of which are clean because they already have all the policies and processes in place before the roots are generated. They should also build and use a parallel testing hierarchy, so that major operations done on the production PKI are done right, first time. Once they have generated new-new roots and intermediates, and got clean audits, they can re-re-apply for inclusion. No-one should be allowed to cross-sign this new hierarchy until,