Re: FW: StartCom inclusion request: next steps
On Tue, Sep 19, 2017 at 3:09 PM, Nick Lamb via dev-security-policywrote: > I have no doubt that this was obvious to people who have worked for a public > CA, but it wasn't obvious to me, so thank you for answering. I think these > answers give us good reason to be confident that a cross-signed certificate > in this situation would not be available to either end subscribers or > StartCom unless/ until the CA which cross-signed it wanted that to happen. > > It might still make sense for Mozilla to clarify that this isn't a good idea, > or even outright forbid it anyway, but I agree with your perspective that > this seemed permissible under the rules as you understood them and wasn't > obviously unreasonable. I'm pretty sure it's already forbidden, since policy version 2.5 anyway (has effective date after the Certinomis shenanigans though): "The CA with a certificate included in Mozilla’s root program MUST disclose this information within a week of certificate creation, and before any such subordinate CA is allowed to issue certificates." 2.5 added the "within a week of certificate creation" [1] . "Creation" vs "My Safe", "Creation" wins. :-) [1] https://github.com/mozilla/pkipolicy/commit/b7d1b6c04458114fbe73fa3f146ad401235c2a1b ___ dev-security-policy mailing list dev-security-policy@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-security-policy
Re: FW: StartCom inclusion request: next steps
Hi Inigo, On 15/09/17 17:30, Inigo Barreira wrote: > There wasn´t a lack of integrity and monitoring, of course not. All PKI logs > were and are signed, it´s just the auditors wanted to add the integrity to > other systems which is not so clear that should have this enabled. For > example, if you want to archive database information for not managing a big > one, the integrity of the logs could be a problem when trying to "move" to an > archive system. I had some discussions about the "scope" of the integrity. > Regarding the monitoring, well, we monitor many things, in both data centers, > 24x7, etc. For this specific issue, it´s true that we didn´t have it > automatically but manually, but well, and we implement a solution, but this > is not a lack of monitoring. I think the audits are to correct and improve > the systems and don´t think any CA at the first time had everything correct. > So, for example, I thought this finding was good because made us improve. > >> Repairing them afterwards does not remove the uncertainty. > > Well, then any issue that you could find, even repaired or fixed, does not > provide you any security and hence you should not trust anyone. Not so. There is particular concern about issues with auditing and monitoring. For other issues, you can check the logs to see whether a particular bug was abused. If the auditing or monitoring is broken or inadequate, you can't tell what happened. >> I may have made a mistake here. I was under the impression that you had >> told me that your new hierarchy, cross-signed by Certnomis, had issued >> 50,000 certificates. Did I misunderstand? If so, my apologies. > > No, or not totally. We have issued those certs but not cross-signed by > Certinomis because we didn´t have the cross-signed certificates so, all of > them were issued under the new startcom hierarchy But once the cross-signed cert is publicly available (and it is; it's in CT, however it got there), all of those certificates become trusted (or potentially trusted, if the owner reconfigures their webserver to serve the intermediate, or if Firefox has already encountered it in the current browsing session). > This is something I don´t understand. Why do you say the audits I presented > don´t meet the BRs? Because of the findings? The auditors indicate those were > fixed I don't believe there's a formal way for an auditor to bindingly say "by the way, the problems we found have since been fixed" in an audit report. But to help me understand: exactly what statement on what page of your audit report(s) are you referring to here? > About the remediation steps, well, I answered the bug about it providing all > the info and yes, you haven´t answered yet nor to approve nor to deny. Right. So why are you proceeding? You might reasonably complain it's taken us a while to respond to that comment about the steps. Yes, it has. The Mozilla inclusion process is slow. :-( >>> In fact, recently, I asked for permission to use the Certinomis cross-signed >> certificates and have no response. I don´t know if this is an administrative >> silence which may allow me to use it but until having a clear direction we >> haven´t used it. >> >> Can you remind me how you asked and when? > > It was in an email of sept 4th, titled "StartCom communication" in which at > the end of the long email I asked for feedback to use the cross-signed > certificates and give additional explanations I have no record of any email with that title, or any email from you between 15th August ("Re: Problem Reporting Mechanism") and 11th September ("Re: Remove old Startcom roots from NSS"). Where did you send it? Gerv ___ dev-security-policy mailing list dev-security-policy@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-security-policy
Re: FW: StartCom inclusion request: next steps
Hi Franck, On 18/09/17 15:49, Franck Leroy wrote: > Our understanding in April was that as long as StartCom is not > allowed by Certinomis to issue EE certs, the disclosure was not > mandated immediately. I think that we need to establish a timeline of the exact events involved here. But I would say that it seems to me that Startcom _were_ issuing EE certs at that time, from the part of their hierarchy that you had cross-signed. In what way was Certnomis forbidding them from doing so? My understanding is that your answer to this question is... > This control that StartCom was not allowed to use our path was > technical in place by the fact that I was the only one to have the > intermediate cross signed certificates, stored (retained) in my > personal safe. that you had not given Startcom a copy of the cross-sign. However, leaving aside for the moment the reasonable question about how such an assertion can be audited, the point is that once the certificate _does_ become public, all of the existing certificates immediately become publicly trusted. Wouldn't you agree? Gerv ___ dev-security-policy mailing list dev-security-policy@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-security-policy
Re: FW: StartCom inclusion request: next steps
On 15/09/17 15:35, Inigo Barreira wrote: > No, those weren´t tests. We allowed the use of curves permitted by the BRs > but this issue came up in the mozilla policy (I think Arkadiusz posted) and I > also asked about it in the last CABF F2F (I asked Ryan about it) and then, > with that outcome and as the browsers didn´t accept them, we revoked and then > not allow the issuance. I think the discussion is still active (i.e. the use > of P-521). Unless Mozilla says otherwise (which we do occasionally), you should not be basing your practice on what we are discussing, or even on draft versions of the policy, but on the published version - which clearly prohibits P-521. Gerv ___ dev-security-policy mailing list dev-security-policy@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-security-policy
Re: FW: StartCom inclusion request: next steps
Le lundi 18 septembre 2017 14:52:27 UTC+2, Ryan Sleevi a écrit : > On Mon, Sep 18, 2017 at 8:12 AM, Inigo Barreira <> > wrote: > Then they misissued a CA certificate and failed to disclose it, and we > should start an incident report into it. Hello In April 2017 the mozilla policy in force (v2.4) stated: “The CA with a certificate included in Mozilla’s CA Certificate Program MUST disclose this information before any such subordinate CA is allowed to issue certificates.” Our understanding in April was that as long as StartCom is not allowed by Certinomis to issue EE certs, the disclosure was not mandated immediately. This control that StartCom was not allowed to use our path was technical in place by the fact that I was the only one to have the intermediate cross signed certificates, stored (retained) in my personal safe. As soon as Certinomis has authorized StartCom to use the path to our root, I disclosed the certificates with the audit reports in the CCADB, and send the certificates to Inigo. May be I misunderstood the Mozilla requirements v2.4, and as I already said in previous post, I do apologize for it. But it was not my intention not to enforce the policy; I personally took care that StartCom could not be able to use the path to our root until a full BR audit assessment report was provided. Regards Franck Leroy ___ dev-security-policy mailing list dev-security-policy@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-security-policy
Re: FW: StartCom inclusion request: next steps
On Mon, Sep 18, 2017 at 8:12 AM, Inigo Barreirawrote: > > We are not seeking to identify personal blame. We are seeking to > understand what, if any, improvements have been made to address such > issues. In reading this thread, I have difficulty finding any discussion > about the steps that StartCom has proposed to improve its awareness of the > expectations placed upon it as a potential participant in the Mozilla > store. Regardless of who bears responsibility for that, the absence of a > robust process - and, unfortunately, the absence of a deep understanding - > does mean that the restablishing of trust can present a significant risk to > the community. > > > > I think I´ve posted everything we did to improve our systems. I replied to > every error posted in the crt.sh explaining what happened and what we did > to fix it for not having the same issue again, but will try to recap here > again. > > > > - Test certificates. We issued test certificates in production > to test the CT behaviour. After the checking those certs were revoked, > within minutes. This was due to an incorrect configuration in the EJBCA > roles that was changed and updated accordingly for not allowing anyone to > issue certs from the EJBCA directly > > - Use of unallowed curves. We issued certificates with P-521 > which is not allowed by Mozilla. We revoked all those certs and configure > the system to not allow it. This remediation was put into production on > mid-july not issuing certs with that curve. > > - RSA parameter not included. We issued one certificate which no > RSA parameter included. We revoked that certificate and started an > investigation. The EJBCA system didn´t check the keys, concretely for this > issue. We developed a solution to check the CSR files properly before > sending to sign > > - Country code not allowed. We issued one certificate with > country code ZR for Zaire, which does not exist officially. We revoked the > cert and checked our internal country code database with the ISO one. We > made the correspondent changes. The cert was reissued with the right code > representing the Democratic Republic of Congo. > > > > Furthermore, we have added x509lint and cablint to our issuance process. > We have integrated crt.sh tool into our CMS system. We have developed a CSR > checking tool. We have updated the EJBCA system to the latest patch, > 6.0.9.5 which also came with a Key (RSA and ECC) validator, and we are also > willing to integrate the zlint once is getting more stable. We have applied > all these tools and we are not misissuing certificates. > Unfortunately, I am not sure how to more effectively communicate that this pattern of issues indicates an organization failure in the review of, application of, and implementation of the Baseline Requirements. Through both coding practices and issuing practices, security and compliance are not being responded to as systemic objectives, but rather as 'one offs', giving the impression of 'whack-a-mole' and ad-hoc response. For example, the country code failure indicates a more deeper failing - did you misunderstand the BRs? Were they not reviewed? Was the code simply not implemented correctly? With the RSA parameters, it similarly indicates a lack of attention. I greatly appreciate the use of and deployment of x509lint and cablint, but those merely offer technical checking that, as an aspiring trusted root CA, you should have already been implementing - whether your own or using those available COTS. The continued approach to issue-and-revoke rather than holistically review the practices and take every step possible to ensure compliance - particularly at a CA that was previously distrusted due to non-compliance - is a particularly egregious oversight that hasn't been responded to. In every response, it still feels as if you're suggesting these are one-offs and coding errors, when the concern being raised is how deeply indicative they are of systemic failures from top to bottom, from policy to technology, from oversight to implementation. Rather than demonstrate how beyond reproach StartCom is, it feels like an excessive emphasis is being put on the ineffective revocation of these certificates, while ignoring the issues that lead to them being issued in the first place - both from policy and from code. It´s not like disagreements, but the example was about a root certificate > private key in a USB stick, so IMO that example starts with a very > problematic issue, because it´s about a root private key and in a USB stick > left in the table, while the issues Startcom did was about end entity > certificates, and nothing related to private keys and not in the root, > that´s what I meant with “quality”. > I understand what you meant. I'm suggesting the community has a perception that the issues StartCom is presently being faced with is as egregious and as serious. I understand you disagree it's as
Re: FW: StartCom inclusion request: next steps
I didn't understand the original below comment by StartCom very well about the cross-sign, but after Ryan's message I understand it better in retrospect: > On Thu, Sep 14, 2017 at 11:05 AM, Inigo Barreira via dev-security-policy < dev-security-policy@lists.mozilla.org> wrote: > > I´ve never said this. In fact, despite having that cross-signed which were provided to us in july we have never used and provided to any of our customers to build a trusted path. So none of those 5, or the new ones, go with the Certinomis path because none have it. But all those 5 certs are untrusted because we´re not in the Mozilla root, not the new one, and the old one was distrusted. > > In fact, recently, I asked for permission to use the Certinomis cross-signed certificates and have no response. I don´t know if this is an administrative silence which may allow me to use it but until having a clear direction we haven´t used it. So this appears to be saying that "all those 5 certs are untrusted" because StartCom didn't provide the full chain to customers, even though such a chain could be constructed. The cross-signature wasn't published in CT until August 2nd, but that's not any sort of guarantee that the cross-signature wasn't discoverable by other means -- its availability until August 2nd is a function of actions by Certinomis that are not disclosed. The August 2nd date is also after StartCom's actions were being publicly questioned, so it suggests the possibility that the cross-signature would have been kept secret for longer, and was only submitted to CT once scrutiny had increased. Whether the cross-cert was issued before the audit report date is also a mystery, especially if it's possible that either Certinomis or StartCom was operating under the assumption that the cross-signature is irrelevant until "delivered" to customers. StartCom has remarked several times in this thread that they are being treated unfairly, but I can think of at least one comparison to a previous distrust event, which is that one of the more significant (in my opinion) issues with Symantec's now-deprecated PKI is that there existed chains that brought U.S. Federal PKI certificates into being trusted by Mozilla. Those chains were, as far as I know, never delivered proactively to customers, but could easily be constructed by any interested party with sufficient knowledge of the universe of cross-signatures. For example, Qualys' SSL Labs reports would automatically construct those chains for sites using FPKI certs, and let users download the full chain in one click. The threat model here is not what ordinary inexpert customers do, but what opportunities an adversary has available to them among the universe of trusted CAs to obtain certificates. In the Symantec/FPKI case, the problem was that an adversary could easily use an FPKI certificate to intercept connections made by Mozilla products, whether or not Symantec or the FPKI ever advertised or proactively enabled this use case. What made this such a big issue, in addition to the scope of the technical impact, is that the issue was not noticed or elevated for years, during which multiple "generations" of cross-signs had been issued and expired. It brought Symantec's ability to understand their own PKI into serious question. So I think the biggest issue here is not so much the technical impact, but that StartCom was communicating inaccurate information to Mozilla. The certs were publicly trusted by Certinomis, whether the cross-signature was delivered to StartCom or to customers or to no one. While presumably this inaccuracy was unintentional, it was enough to cause Gerv to express public confusion and doubt about whether the certificates were part of the cross-signed hierarchy. It also reflects a potentially dangerous difference of perspective between StartCom and root stores in how StartCom evaluates the trust and impact of the certificates they issue. For a CA that has been operating for as long as StartCom has, I think it's fair to describe this as concerning. I also think that Certinomis, whose cross-signing practices are now being scrutinized, should proactively post to this list with a timeline of its own actions during this process, so that their actions can be understood in the context of StartCom's. -- Eric ___ dev-security-policy mailing list dev-security-policy@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-security-policy
Re: FW: StartCom inclusion request: next steps
On Fri, Sep 15, 2017 at 12:30 PM, Inigo Barreira via dev-security-policy < dev-security-policy@lists.mozilla.org> wrote: > > > > Hi Inigo, > > > > On 14/09/17 16:05, Inigo Barreira wrote: > > > Those tests were done to check the CT behaviour, there was any other > > testing of the new systems, just for the CT. > > > > Is there any reason those tests could not have been done using a parallel > > testing hierarchy (other than the fact that you hadn't set one up)? > > I think I provided the reasons. We were distrusted, not re-applied yet, > those certs lived for minutes, ... Can you point where the Baseline Requirements and/or Root Store policies exclude "certs lived for minutes" from being in scope? Or where revocation absolves a CA of the issuance in the first place? > So, I think these are the reasons. It´s not an excuse and we didn´t expect > this maremágnum. I believe the lack of expectation is precisely why there is significant concern here. CAs are expected to be global stewards of trust, operating beyond reproach - which generally means assuming all things are forbidden unless expressly permitted, and even when it seems they _may_ be permitted, to consider the full implications of the interpretation and to check if there are multiple interpretations. > There´s been long discussions about the harm long term certificates can > make and asked CAs about short term to avoid damages. These certs, that > it´s true was a mistake, only lived for minutes, so I don´t know what else > I can add to this matter. > I think you've added a lot, and while the engagement has been valuable, hopefully this clarifies precisely why these responses have been so deeply concerning towards demonstrating trustworthiness. > > But it is the job of a CA to be aware of browser policies. > > Yes, you´re right. And it was my fault for not have checked in deep this > particular one. > We are not seeking to identify personal blame. We are seeking to understand what, if any, improvements have been made to address such issues. In reading this thread, I have difficulty finding any discussion about the steps that StartCom has proposed to improve its awareness of the expectations placed upon it as a potential participant in the Mozilla store. Regardless of who bears responsibility for that, the absence of a robust process - and, unfortunately, the absence of a deep understanding - does mean that the restablishing of trust can present a significant risk to the community. > > I think lack of monitoring and lack of integrity of logs are serious > issues. > > There wasn´t a lack of integrity and monitoring, of course not. All PKI > logs were and are signed, it´s just the auditors wanted to add the > integrity to other systems which is not so clear that should have this > enabled. For example, if you want to archive database information for not > managing a big one, the integrity of the logs could be a problem when > trying to "move" to an archive system. I had some discussions about the > "scope" of the integrity. I am wholly uncertain how to interpret what you're saying here. > Regarding the monitoring, well, we monitor many things, in both data > centers, 24x7, etc. For this specific issue, it´s true that we didn´t have > it automatically but manually, but well, and we implement a solution, but > this is not a lack of monitoring. I think the audits are to correct and > improve the systems and don´t think any CA at the first time had everything > correct. So, for example, I thought this finding was good because made us > improve. > I agree that a well-executed audit can help a CA identify areas of improvement. However, a well-executed audit can also identify issues of non-compliance or identify issues of risk that the community may find unacceptable, independent of the auditors own assessment. > > Repairing them afterwards does not remove the uncertainty. > > Well, then any issue that you could find, even repaired or fixed, does not > provide you any security and hence you should not trust anyone. > I do not think this demonstrates a positive awareness of the issues being discussed. Again, as CAs are expected to be stewards of global trust, it is expected that CAs seek to both individually improve and rise above 'the minimum' requirements, and to seek ways to improve those minimums. > If you said "I left the root certificate private key on a USB stick on > the desk in my unlocked > > office over the weekend - but it's OK, I've remediated the problem now, > and > > it's back in the safe", that would not remove the uncertainty about > whether > > someone had done something with it in the mean time. > > Well, I don´t think this is the same "quality" of example. > It is useful to know what you think, as it highlights disagreement. However, it's more useful to know why you think this. I would suggest that, for most people examining both the information provided on the issues, the facts, and the discussion so far, it is very
RE: FW: StartCom inclusion request: next steps
> > Hi Inigo, > > On 14/09/17 16:05, Inigo Barreira wrote: > > Those tests were done to check the CT behaviour, there was any other > testing of the new systems, just for the CT. > > Is there any reason those tests could not have been done using a parallel > testing hierarchy (other than the fact that you hadn't set one up)? I think I provided the reasons. We were distrusted, not re-applied yet, those certs lived for minutes, ... So, I think these are the reasons. It´s not an excuse and we didn´t expect this maremágnum. There´s been long discussions about the harm long term certificates can make and asked CAs about short term to avoid damages. These certs, that it´s true was a mistake, only lived for minutes, so I don´t know what else I can add to this matter. > > > Some of these "mis-issuances" were due to some incongruencies between > > the BRs and the Mozilla policy, such as the use of different curves > > (allowed by the BRs but not for some browsers), > > But it is the job of a CA to be aware of browser policies. Yes, you´re right. And it was my fault for not have checked in deep this particular one. > > > As far as I know, the current manager of Startcom has not been previously > accused of deception or bad action. > > No, indeed. There is no accusation that you have been deceptive towards > anyone. > > > Yes, it´s not a policy violation. As explained, this was a problem in the > > EJBCA > with the UTF8 encoding. > > That was the reason for the revocation; it's not the reason for the key reuse. Yes, right, but this is allowed afaik. It´s mentioned in the BRs. > > >> * The WT/BR/EV audits on StartCom's website are significantly > >> qualified, and they include lack of controls on issuance. They should > >> have clean ones done before we permit any inclusion request to proceed. > The qualifications include: > >> > >> - Risk analysis process defined but not implemented > >> - Business continuity plan defined but not implemented > >> - Audit logs not guaranteed to have integrity > >> - Monitoring system cannot detect security-related changes to > >> Certificate Systems > > > > Yes, this is also true. Our webtrust audits have findings but those are not > > so > significant according to the auditors who signed the reports, so I assume the > auditors thought that the system is good enough to have the audit report in > place. > > I think lack of monitoring and lack of integrity of logs are serious issues. There wasn´t a lack of integrity and monitoring, of course not. All PKI logs were and are signed, it´s just the auditors wanted to add the integrity to other systems which is not so clear that should have this enabled. For example, if you want to archive database information for not managing a big one, the integrity of the logs could be a problem when trying to "move" to an archive system. I had some discussions about the "scope" of the integrity. Regarding the monitoring, well, we monitor many things, in both data centers, 24x7, etc. For this specific issue, it´s true that we didn´t have it automatically but manually, but well, and we implement a solution, but this is not a lack of monitoring. I think the audits are to correct and improve the systems and don´t think any CA at the first time had everything correct. So, for example, I thought this finding was good because made us improve. > Repairing them afterwards does not remove the uncertainty. Well, then any issue that you could find, even repaired or fixed, does not provide you any security and hence you should not trust anyone. > If you said "I left the root certificate private key on a USB stick on the > desk in my unlocked > office over the weekend - but it's OK, I've remediated the problem now, and > it's back in the safe", that would not remove the uncertainty about whether > someone had done something with it in the mean time. Well, I don´t think this is the same "quality" of example. > > > I don´t know how these you mention have applied, but I remember lots of > issues regarding Google and the acquisition of the Globalsign roots and how > they proceeded. > > I think "lots of issues" is an overstatement. There was an issue regarding the > scope of their audits which was raised publicly, but it was something that > Google has explicitly sought our approval for in advance. Yes, it´s an overstatement but similar to others. I said that I didn´t know what or how they had applied but you put them as examples and just wanted to indicate that there were discussions, so maybe was not so smooth, but again, I don´t know. > > > Not sure about this. We were distrusted in october 2016, the new system > started to operate in april 2017, which is not related to the old one which > has > been switched off. None of the new certs are trusted in Mozilla Firefox and > we notify our users so by messages in the web and applications. > > I may have made a mistake here. I was under the impression that you had >
Re: FW: StartCom inclusion request: next steps
I'm fairly confused by your answers, if the only thing you tested in production was CT, why was the system issuing non-compliant certs? Why did production CT testing come before having established, tested, and verified a compliant certificate profile? Alex On Fri, Sep 15, 2017 at 10:35 AM, Inigo Barreira via dev-security-policy < dev-security-policy@lists.mozilla.org> wrote: > > On 15/09/17 11:01, Inigo Barreira wrote: > > > Considering that we were distrusted, that we didn´t reapply for > > > inclussion, that CT is only required by Chrome and it´s not included > > > in the Mozilla policy (even we were requested that all of our certs > > > had to be CT logged) nor required by Firefox, that those certs were > > > under our control all the time and lived for some minutes because were > > > revoked inmediately, at that time, when we did it, we didn´t expect > > > this reaction for sure. > > > > But surely CT testing is not the only sort of testing you've been doing? > > Yes, this is the only test we did it in production > > > E.g. you made some test certificates with different types of ECC curve, > which > > you then had to revoke some of as against browser policies. > > No, those weren´t tests. We allowed the use of curves permitted by the BRs > but this issue came up in the mozilla policy (I think Arkadiusz posted) and > I also asked about it in the last CABF F2F (I asked Ryan about it) and > then, with that outcome and as the browsers didn´t accept them, we revoked > and then not allow the issuance. I think the discussion is still active > (i.e. the use of P-521). > > > If these had been in a testing hierarchy there would have been no > problem. > > > > CAs have been heavily criticised over the past few years for issuing test > > certificates in public hierarchies (see e.g. Symantec). The danger of > doing so > > should be well known to all CAs by now. > > Yes, I know. But the only testing we did in production was the one related > to the CT. > > > > Perhaps once a test has been passed and checked in a testing system, and > if > > the certificates concerned do not violate any policies, it could be > repeated on > > a production system to deal with any possible differences between the > two. > > But starting with the production system is not a good idea. > > True, but it seems you´re understanding that we have only a production > system in which we test everything and this is not the case. Before moving > anything into production, we have tested in development and in the QA > system. > > > > Gerv > > ___ > dev-security-policy mailing list > dev-security-policy@lists.mozilla.org > https://lists.mozilla.org/listinfo/dev-security-policy > > ___ dev-security-policy mailing list dev-security-policy@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-security-policy
Re: FW: StartCom inclusion request: next steps
On 15/09/17 09:24, Inigo Barreira wrote: > AFAIK, Certinomis only disclosed in the CCADB That means it's published and available. As noted in my other reply, information as to exactly what this cross-sign enables trust for would be most helpful, as I may have misunderstood previous statements on this topic. Gerv ___ dev-security-policy mailing list dev-security-policy@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-security-policy
Re: FW: StartCom inclusion request: next steps
Hi Inigo, On 14/09/17 16:05, Inigo Barreira wrote: > Those tests were done to check the CT behaviour, there was any other testing > of the new systems, just for the CT. Is there any reason those tests could not have been done using a parallel testing hierarchy (other than the fact that you hadn't set one up)? > Some of these "mis-issuances" were due to some incongruencies between the BRs > and the Mozilla policy, such as the use of different curves (allowed by the > BRs but not for some browsers), But it is the job of a CA to be aware of browser policies. > As far as I know, the current manager of Startcom has not been previously > accused of deception or bad action. No, indeed. There is no accusation that you have been deceptive towards anyone. > Yes, it´s not a policy violation. As explained, this was a problem in the > EJBCA with the UTF8 encoding. That was the reason for the revocation; it's not the reason for the key reuse. >> * The WT/BR/EV audits on StartCom's website are significantly qualified, and >> they include lack of controls on issuance. They should have clean ones done >> before we permit any inclusion request to proceed. The qualifications >> include: >> >> - Risk analysis process defined but not implemented >> - Business continuity plan defined but not implemented >> - Audit logs not guaranteed to have integrity >> - Monitoring system cannot detect security-related changes to >> Certificate Systems > > Yes, this is also true. Our webtrust audits have findings but those are not > so significant according to the auditors who signed the reports, so I assume > the auditors thought that the system is good enough to have the audit report > in place. I think lack of monitoring and lack of integrity of logs are serious issues. Repairing them afterwards does not remove the uncertainty. If you said "I left the root certificate private key on a USB stick on the desk in my unlocked office over the weekend - but it's OK, I've remediated the problem now, and it's back in the safe", that would not remove the uncertainty about whether someone had done something with it in the mean time. > I don´t know how these you mention have applied, but I remember lots of > issues regarding Google and the acquisition of the Globalsign roots and how > they proceeded. I think "lots of issues" is an overstatement. There was an issue regarding the scope of their audits which was raised publicly, but it was something that Google has explicitly sought our approval for in advance. > Not sure about this. We were distrusted in october 2016, the new system > started to operate in april 2017, which is not related to the old one which > has been switched off. None of the new certs are trusted in Mozilla Firefox > and we notify our users so by messages in the web and applications. I may have made a mistake here. I was under the impression that you had told me that your new hierarchy, cross-signed by Certnomis, had issued 50,000 certificates. Did I misunderstand? If so, my apologies. > Ok, I see this is a new requirement that was not imposed last time in which > you recommended and allowed us to be cross-signed as many other CAs have done > in the past to be in the business. Mozilla did not recommend that you be cross-signed. Certnomis raised the possibility, and we said that it could happen when you had met the BRs and completed the remediation steps. The plan we "approved" was not a cross-signing plan. The audits show you haven't yet met the BRs, and we have not yet signed off on you having completed the remediation steps. > We´ve met all the conditions, new system, new management, security audit and > webtrust audit and CT logging. In those conditions, it was not mentioned that > the webtrust audit should be clean Isn't that implied? > I´ve never said this. In fact, despite having that cross-signed which were > provided to us in july we have never used and provided to any of our > customers to build a trusted path. So none of those 5, or the new ones, > go with the Certinomis path because none have it. But all those 5 certs > are untrusted because we´re not in the Mozilla root, not the new one, and the > old one was distrusted. So the 50,000 certs you mentioned are issued in your old hierarchy, which is not cross-signed by Certnomis? If you could clarify the numbers for your old and new PKI, and confirm that the Certnomis cross-signs apply only to the new one, that would be helpful. Again, apologies for any misunderstanding on my part. > In fact, recently, I asked for permission to use the Certinomis cross-signed > certificates and have no response. I don´t know if this is an administrative > silence which may allow me to use it but until having a clear direction we > haven´t used it. Can you remind me how you asked and when? > I think this has been explained. I don´t understand why you say it´s a "poor > quality PHP code". Because poor quality PHP code does
RE: FW: StartCom inclusion request: next steps
> Hi Inigo, > > To add from the last post. > > I know this is unwelcome news to you but I feel that with all these incidents > happening right now with Symantec and the incidents before, we can't really > take any more chances. Every incident is eroding trust in this system and if > we > want more people to take up adoption of https in the long term, I feel that > we need to start operating infrastructure above board and without issues. > > James James, I share with you this desire. But we´re seeing one day and another that issues are still happening, and not all are from StartCom :-) so to avoid eroding trust what do you suggest if not give more chances? Remove them all? > ___ > dev-security-policy mailing list > dev-security-policy@lists.mozilla.org > https://lists.mozilla.org/listinfo/dev-security-policy smime.p7s Description: S/MIME cryptographic signature ___ dev-security-policy mailing list dev-security-policy@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-security-policy
RE: FW: StartCom inclusion request: next steps
> > > > > > > Those tests were done to check the CT behaviour, there was any > > > > other > > > testing of the new systems, just for the CT. Those certs were under > > > control all the time and were lived for some minutes because were > > > revoked inmediately after checking the certs were logged correctly > > > in the CTs. It´s not a mis- issuance by means of we didn´t know what > > > happened, we had to investigate, etc. It was not a good practice and > > > I can´t excuse for that, but it was not related to the regular > > > issuance procedure as someone suggested. We provided a report in > > > which indicated all that happened and what we did to not happen this > again, updating the EJBCA roles permissions. > > > > > > 1) Why didn't StartCom build a test hierarchy? > > > > Considering that we were distrusted, that we didn´t reapply for inclussion, > that CT is only required by Chrome and it´s not included in the Mozilla policy > (even we were requested that all of our certs had to be CT logged) nor > required by Firefox, that those certs were under our control all the time and > lived for some minutes because were revoked inmediately, at that time, when > we did it, we didn´t expect this reaction for sure. > > > > Of course if we had known it we hadn´t done it and for sure had built > > a test hierarchy but there´s nothing we can do now. Only wanted to > > state that those certs were under our control all the time, and lived > > for some minutes because were revoked after the test. There were not > > any other testing of any other nature directly in the production > > system > > > > > 2) Why didn't StartCom use the TestTube CT log for testing CT? > > > > We tried to check and test the same behaviour before going live with the CT > logging, so followed the requirements to use 3 logs, one google and one non- > google, for the EVs and this is what we did. We used the same settings we > had before the distrust using the startcom log and the google ones (pilot and > rocketeer). > > Hi Inigo, > > I'm just trying to get everything straight, cleared up and then we can move > on. > > I found all of your answers very concerning in the way you've conducted > testing. I thought that all CAs must have some type of test hierarchy in place > to test new software, Well, this is not required. Of course we have a development and a testing system, which are not "connected" into production. As we wanted to check the same behaviour for the CT ecosystem than before the distrust, we did it directly in production, but don´t take this as we didn´t test it before in our development and testing systems, for sure we did. > requirements and etc before going live but from the answers you've given, > StartCom neglected this in favour to test as you go > along and deal with the problems using a live CA hierarchy. As said, this wasn´t a "live" CA hierarchy. We were distrusted, nothing issued from that "live" CA was trusted, this wasn´t any harm to any of the Mozilla users. > All this feels very amateurish and doesn't give me the any confidence at all > that your CA is > proven itself to be a trustworthy part of this infrastructure. I respect this opinion but can´t agree with it. > > I recommend to Mozilla to require StartCom to start again fresh. > > 1. Build a test hierarchy. Test the software and etc to industry guidelines. > 2. Build a new production hierarchy (including new HSMs, keys, roots, etc.) > and then re-apply for inclusion into the Mozilla root program. > 3. Once approved then you can cross-sign your roots with another CA. > > While this is happening. You can resell certificates from other CAs and build > up trust in the industry which will benefit you in the long term I feel. > > James > ___ > dev-security-policy mailing list > dev-security-policy@lists.mozilla.org > https://lists.mozilla.org/listinfo/dev-security-policy smime.p7s Description: S/MIME cryptographic signature ___ dev-security-policy mailing list dev-security-policy@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-security-policy
Re: FW: StartCom inclusion request: next steps
On Friday, September 15, 2017 at 12:30:00 PM UTC+1, James Burton wrote: > On Friday, September 15, 2017 at 10:56:11 AM UTC+1, Inigo Barreira wrote: > > > > > > > Those tests were done to check the CT behaviour, there was any other > > > testing of the new systems, just for the CT. Those certs were under > > > control all > > > the time and were lived for some minutes because were revoked inmediately > > > after checking the certs were logged correctly in the CTs. It´s not a mis- > > > issuance by means of we didn´t know what happened, we had to investigate, > > > etc. It was not a good practice and I can´t excuse for that, but it was > > > not > > > related to the regular issuance procedure as someone suggested. We > > > provided a report in which indicated all that happened and what we did to > > > not happen this again, updating the EJBCA roles permissions. > > > > > > 1) Why didn't StartCom build a test hierarchy? > > > > Considering that we were distrusted, that we didn´t reapply for inclussion, > > that CT is only required by Chrome and it´s not included in the Mozilla > > policy (even we were requested that all of our certs had to be CT logged) > > nor required by Firefox, that those certs were under our control all the > > time and lived for some minutes because were revoked inmediately, at that > > time, when we did it, we didn´t expect this reaction for sure. > > > > Of course if we had known it we hadn´t done it and for sure had built a > > test hierarchy but there´s nothing we can do now. Only wanted to state that > > those certs were under our control all the time, and lived for some minutes > > because were revoked after the test. There were not any other testing of > > any other nature directly in the production system > > > > > 2) Why didn't StartCom use the TestTube CT log for testing CT? > > > > We tried to check and test the same behaviour before going live with the CT > > logging, so followed the requirements to use 3 logs, one google and one > > non-google, for the EVs and this is what we did. We used the same settings > > we had before the distrust using the startcom log and the google ones > > (pilot and rocketeer). > > Hi Inigo, > > I'm just trying to get everything straight, cleared up and then we can move > on. > > I found all of your answers very concerning in the way you've conducted > testing. I thought that all CAs must have some type of test hierarchy in > place to test new software, requirements and etc before going live but from > the answers you've given, StartCom neglected this in favour to test as you go > along and deal with the problems using a live CA hierarchy. All this feels > very amateurish and doesn't give me the any confidence at all that your CA is > proven itself to be a trustworthy part of this infrastructure. > > I recommend to Mozilla to require StartCom to start again fresh. > > 1. Build a test hierarchy. Test the software and etc to industry guidelines. > 2. Build a new production hierarchy (including new HSMs, keys, roots, etc.) > and then re-apply for inclusion into the Mozilla root program. > 3. Once approved then you can cross-sign your roots with another CA. > > While this is happening. You can resell certificates from other CAs and build > up trust in the industry which will benefit you in the long term I feel. > > James Hi Inigo, To add from the last post. I know this is unwelcome news to you but I feel that with all these incidents happening right now with Symantec and the incidents before, we can't really take any more chances. Every incident is eroding trust in this system and if we want more people to take up adoption of https in the long term, I feel that we need to start operating infrastructure above board and without issues. James ___ dev-security-policy mailing list dev-security-policy@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-security-policy
Re: FW: StartCom inclusion request: next steps
On Friday, September 15, 2017 at 10:56:11 AM UTC+1, Inigo Barreira wrote: > > > > > Those tests were done to check the CT behaviour, there was any other > > testing of the new systems, just for the CT. Those certs were under control > > all > > the time and were lived for some minutes because were revoked inmediately > > after checking the certs were logged correctly in the CTs. It´s not a mis- > > issuance by means of we didn´t know what happened, we had to investigate, > > etc. It was not a good practice and I can´t excuse for that, but it was not > > related to the regular issuance procedure as someone suggested. We > > provided a report in which indicated all that happened and what we did to > > not happen this again, updating the EJBCA roles permissions. > > > > 1) Why didn't StartCom build a test hierarchy? > > Considering that we were distrusted, that we didn´t reapply for inclussion, > that CT is only required by Chrome and it´s not included in the Mozilla > policy (even we were requested that all of our certs had to be CT logged) nor > required by Firefox, that those certs were under our control all the time and > lived for some minutes because were revoked inmediately, at that time, when > we did it, we didn´t expect this reaction for sure. > > Of course if we had known it we hadn´t done it and for sure had built a test > hierarchy but there´s nothing we can do now. Only wanted to state that those > certs were under our control all the time, and lived for some minutes because > were revoked after the test. There were not any other testing of any other > nature directly in the production system > > > 2) Why didn't StartCom use the TestTube CT log for testing CT? > > We tried to check and test the same behaviour before going live with the CT > logging, so followed the requirements to use 3 logs, one google and one > non-google, for the EVs and this is what we did. We used the same settings we > had before the distrust using the startcom log and the google ones (pilot and > rocketeer). Hi Inigo, I'm just trying to get everything straight, cleared up and then we can move on. I found all of your answers very concerning in the way you've conducted testing. I thought that all CAs must have some type of test hierarchy in place to test new software, requirements and etc before going live but from the answers you've given, StartCom neglected this in favour to test as you go along and deal with the problems using a live CA hierarchy. All this feels very amateurish and doesn't give me the any confidence at all that your CA is proven itself to be a trustworthy part of this infrastructure. I recommend to Mozilla to require StartCom to start again fresh. 1. Build a test hierarchy. Test the software and etc to industry guidelines. 2. Build a new production hierarchy (including new HSMs, keys, roots, etc.) and then re-apply for inclusion into the Mozilla root program. 3. Once approved then you can cross-sign your roots with another CA. While this is happening. You can resell certificates from other CAs and build up trust in the industry which will benefit you in the long term I feel. James ___ dev-security-policy mailing list dev-security-policy@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-security-policy
RE: FW: StartCom inclusion request: next steps
> > > Those tests were done to check the CT behaviour, there was any other > testing of the new systems, just for the CT. Those certs were under control > all > the time and were lived for some minutes because were revoked inmediately > after checking the certs were logged correctly in the CTs. It´s not a mis- > issuance by means of we didn´t know what happened, we had to investigate, > etc. It was not a good practice and I can´t excuse for that, but it was not > related to the regular issuance procedure as someone suggested. We > provided a report in which indicated all that happened and what we did to > not happen this again, updating the EJBCA roles permissions. > > 1) Why didn't StartCom build a test hierarchy? Considering that we were distrusted, that we didn´t reapply for inclussion, that CT is only required by Chrome and it´s not included in the Mozilla policy (even we were requested that all of our certs had to be CT logged) nor required by Firefox, that those certs were under our control all the time and lived for some minutes because were revoked inmediately, at that time, when we did it, we didn´t expect this reaction for sure. Of course if we had known it we hadn´t done it and for sure had built a test hierarchy but there´s nothing we can do now. Only wanted to state that those certs were under our control all the time, and lived for some minutes because were revoked after the test. There were not any other testing of any other nature directly in the production system > 2) Why didn't StartCom use the TestTube CT log for testing CT? We tried to check and test the same behaviour before going live with the CT logging, so followed the requirements to use 3 logs, one google and one non-google, for the EVs and this is what we did. We used the same settings we had before the distrust using the startcom log and the google ones (pilot and rocketeer). smime.p7s Description: S/MIME cryptographic signature ___ dev-security-policy mailing list dev-security-policy@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-security-policy
Re: FW: StartCom inclusion request: next steps
> Those tests were done to check the CT behaviour, there was any other testing > of the new systems, just for the CT. Those certs were under control all the > time and were lived for some minutes because were revoked inmediately after > checking the certs were logged correctly in the CTs. It´s not a mis-issuance > by means of we didn´t know what happened, we had to investigate, etc. It was > not a good practice and I can´t excuse for that, but it was not related to > the regular issuance procedure as someone suggested. We provided a report in > which indicated all that happened and what we did to not happen this again, > updating the EJBCA roles permissions. 1) Why didn't StartCom build a test hierarchy? 2) Why didn't StartCom use the TestTube CT log for testing CT? ___ dev-security-policy mailing list dev-security-policy@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-security-policy
RE: FW: StartCom inclusion request: next steps
> On 14/09/2017 17:05, Inigo Barreira wrote: > > All, > > > > ... > >> > >> We should add the existing Certnomis cross-signs to OneCRL to revoke > >> all the existing certificates. As of 10th August (now a month ago) > >> StartCom said they have 5 outstanding SSL certs which are valid > >> due to the Certnomis cross- sign. > > > > I´ve never said this. In fact, despite having that cross-signed which were > provided to us in july we have never used and provided to any of our > customers to build a trusted path. So none of those 5, or the new ones, > go with the Certinomis path because none have it. But all those 5 certs > are untrusted because we´re not in the Mozilla root, not the new one, and the > old one was distrusted. > > In fact, recently, I asked for permission to use the Certinomis cross-signed > certificates and have no response. I don´t know if this is an administrative > silence which may allow me to use it but until having a clear direction we > haven´t used it. > > > > > I can't speak for Mozilla, but the obvious point is, that as soon as the cross > certificate from Certinomis was published in CT-logs or elsewhere, any > StartCom customer could (and still can) download it and install it on their > server, thus activating the Certinomis path for their server. > AFAIK, Certinomis only disclosed in the CCADB and even we received it, we have not sent to any customer nor posted anywhere. And I know crt.sh includes it and when questioned for it Rob answered that crt.sh includes all certs that have been submitted to one or more of the monitored CT logs, and includes all trust paths that can be built. I don´t think any StartCom customer has downloaded from wherever it could be and installed and created the trusted path. Our customers base are not very familiar with this stuff. Usually we need to provide all of them with instructions on what and how to do it. > > Enjoy > > Jakob > -- > Jakob Bohm, CIO, Partner, WiseMo A/S. https://www.wisemo.com > Transformervej 29, 2860 Søborg, Denmark. Direct +45 31 13 16 10 This > public discussion message is non-binding and may contain errors. > WiseMo - Remote Service Management for PCs, Phones and Embedded > ___ > dev-security-policy mailing list > dev-security-policy@lists.mozilla.org > https://lists.mozilla.org/listinfo/dev-security-policy smime.p7s Description: S/MIME cryptographic signature ___ dev-security-policy mailing list dev-security-policy@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-security-policy
RE: FW: StartCom inclusion request: next steps
Yes, there are similar ones everywhere, so I´m familiar with it :-) And you´re right, I also make contributions in many other places, ETSI, ENISA, CABF (used to), ... and not get paid for that, but it´s also true that the way the distrust happened didn´t give us time or much time to act accordingly, even now people is asking for the distrust or asking for refunds due to the distrust. And yes, we tried to do it quickly and we know what happened but we didn´t re-apply at that time. We got issues and until all of them were fixed and we receive the OK we didn´t move forward. We wanted to meet the deadline we imposed ourselves and even reached, it was not in a good shape, so, had to start again. And that´s what I said internally, that it didn´t matter if we could not make it in 6 months as indicated in the sanction and it took us nearly 10, taking into account that everything was new. Best regards Iñigo Barreira CEO StartCom CA Limited > -Original Message- > From: dev-security-policy [mailto:dev-security-policy- > bounces+inigo=startcomca@lists.mozilla.org] On Behalf Of Nick Lamb via > dev-security-policy > Sent: viernes, 15 de septiembre de 2017 1:22 > To: mozilla-dev-security-pol...@lists.mozilla.org > Subject: Re: FW: StartCom inclusion request: next steps > > On Thursday, 14 September 2017 16:00:35 UTC+1, Inigo Barreira wrote: > > Well, finally this is a business and I don´t think none on this list is > > working > for free. At the end everyone has his/her salary, etc. But that was not the > main reason because getting included in the root programs takes time but > wanted to provide our customers which gave us support for what happened > with the distrust (which IMHO in the case of Startcom was very aggressive) a > solution generating a new fresh and clean system. > > I can't speak for other people contributing to m.d.s.policy but of course I am > not paid to do this. I have a job, but it isn't this. I have never asked my > employer's permission to contribute here, judging it to be entirely outside > their purview. My job does include running a (small, private) CA, but then it > also includes maintaining a DBMS, a web crawler and all sorts of other stuff, > I'm sure it would be possible to identify some connection to my job for almost > any technical contribution I could make anywhere. > > There is an proverb in English, I am not sure if you're familiar with it, > "More > haste, less speed". What this means is that in trying to perform a task as > quickly as possible you may instead cause yourself such extra trouble that in > the end the task takes even longer to complete. I am sure that even if you > have not come across a phrase like this before you can see its applicability > to > the situation for StartCom. > ___ > dev-security-policy mailing list > dev-security-policy@lists.mozilla.org > https://lists.mozilla.org/listinfo/dev-security-policy smime.p7s Description: S/MIME cryptographic signature ___ dev-security-policy mailing list dev-security-policy@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-security-policy
Re: FW: StartCom inclusion request: next steps
On 14/09/2017 17:05, Inigo Barreira wrote: All, ... We should add the existing Certnomis cross-signs to OneCRL to revoke all the existing certificates. As of 10th August (now a month ago) StartCom said they have 5 outstanding SSL certs which are valid due to the Certnomis cross- sign. I´ve never said this. In fact, despite having that cross-signed which were provided to us in july we have never used and provided to any of our customers to build a trusted path. So none of those 5, or the new ones, go with the Certinomis path because none have it. But all those 5 certs are untrusted because we´re not in the Mozilla root, not the new one, and the old one was distrusted. In fact, recently, I asked for permission to use the Certinomis cross-signed certificates and have no response. I don´t know if this is an administrative silence which may allow me to use it but until having a clear direction we haven´t used it. I can't speak for Mozilla, but the obvious point is, that as soon as the cross certificate from Certinomis was published in CT-logs or elsewhere, any StartCom customer could (and still can) download it and install it on their server, thus activating the Certinomis path for their server. Enjoy Jakob -- Jakob Bohm, CIO, Partner, WiseMo A/S. https://www.wisemo.com Transformervej 29, 2860 Søborg, Denmark. Direct +45 31 13 16 10 This public discussion message is non-binding and may contain errors. WiseMo - Remote Service Management for PCs, Phones and Embedded ___ dev-security-policy mailing list dev-security-policy@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-security-policy
Re: FW: StartCom inclusion request: next steps
On Thursday, 14 September 2017 16:00:35 UTC+1, Inigo Barreira wrote: > Well, finally this is a business and I don´t think none on this list is > working for free. At the end everyone has his/her salary, etc. But that was > not the main reason because getting included in the root programs takes time > but wanted to provide our customers which gave us support for what happened > with the distrust (which IMHO in the case of Startcom was very aggressive) a > solution generating a new fresh and clean system. I can't speak for other people contributing to m.d.s.policy but of course I am not paid to do this. I have a job, but it isn't this. I have never asked my employer's permission to contribute here, judging it to be entirely outside their purview. My job does include running a (small, private) CA, but then it also includes maintaining a DBMS, a web crawler and all sorts of other stuff, I'm sure it would be possible to identify some connection to my job for almost any technical contribution I could make anywhere. There is an proverb in English, I am not sure if you're familiar with it, "More haste, less speed". What this means is that in trying to perform a task as quickly as possible you may instead cause yourself such extra trouble that in the end the task takes even longer to complete. I am sure that even if you have not come across a phrase like this before you can see its applicability to the situation for StartCom. ___ dev-security-policy mailing list dev-security-policy@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-security-policy