I summarize the issue in a form of an incident report. 1. How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.
The presence of the doppelganger CA certificates was known since 2009, and it did not cause problems during the operation. The situation was discussed with Mozilla first in November 2016. The problem came up with the “Audit Letter Validation Failures” in 2019 when new functions were introduced in the CCADB. 2019-11-22 Wayne Thayer wrote the Comment 30 to our root inclusion request bug and mentioned the 9 audit letter validation failures in the CCADB. https://bugzilla.mozilla.org/show_bug.cgi?id=1445364#c30 https://groups.google.com/forum/#!msg/mozilla.dev.security.policy/M7NGwCh14DI/9Go4rOTgBgAJ https://wiki.mozilla.org/CA/Audit_Letter_Validation 2. A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done. The following timeline includes the full list of events concerning the doppelganger CA certificates from their issuance to the present day. 2009-03-06 Microsec Ltd. issued the following root certificate: - Microsec e-Szigno Root CA 2009 ( 2370027485 ) Expiry date: 2038-01-18 2009-03-09 Microsec Ltd. issued the following subordinate CA certificates: - Advanced Class 2 e-Szigno CA 2009 ( 2629646531 ) - Advanced Class 3 e-Szigno CA 2009 ( 12726032 ) - Advanced Pseudonymous e-Szigno CA 2009 ( 2629646669 ) - Qualified Pseudonymous e-Szigno CA 2009 ( 2629646659 ) - Qualified e-Szigno CA 2009 ( 2629646584 ) Expiry date of each of these certificates: 2038-01-17 2009-06-16 Microsec reissued the following root certificate: - Microsec e-Szigno Root CA 2009 ( 194998 ) Expiry date: 2029-12-30 In the certificate the following fields were changed: validity notBefore and notAfter dates, and the certificatePolicies extension. Microsec reissued the following subordinate CA certificates: - Advanced Class 2 e-Szigno CA 2009 ( 2629646790 ) - Advanced Class 3 e-Szigno CA 2009 ( 12722207 ) - Advanced Pseudonymous e-Szigno CA 2009 ( 2629646543 ) - Qualified Pseudonymous e-Szigno CA 2009 ( 26312005 ) - Qualified e-Szigno CA 2009 ( 26312004 ) Expiry date of each of these certificates (aligned with the new root): 2029-12-29 In the certificates the following fields were changed: validity notBefore and notAfter dates, and the certificatePolicies extension. 2009-12-02 Microsec reissued the following root certificate: - Microsec e-Szigno Root CA 2009 ( 149444539 ) Expiry date was unchanged: 2029-12-30 The certificatePolicies extension was dropped from the certificate, and the place of the keyUsage extension changed within the ASN.1 structure. Microsec reissued the following subordinate CA certificates: - Advanced Class 2 e-Szigno CA 2009 ( 12625481 ) - Advanced Class 3 e-Szigno CA 2009 ( 194999 ) - Advanced Pseudonymous e-Szigno CA 2009 ( 12716021 ) - Qualified Pseudonymous e-Szigno CA 2009 ( 12716127 ) - Qualified e-Szigno CA 2009 ( 12721748 ) In the certificates the validity notBefore date changed to the issuance date, the certificatePolicies extension changed to anyPolicy, and the URL pointing to the location of CPS documents changed to a uniform value for qualified and non-qualified certificates. In summary, all above certificates were reissued by Microsec two times during 2009. In these two cases each certificate was reissued with an unchanged public key and unchanged Subject DN as compared to the previous corresponding certificate, while some other fields were changed. The CA certificates were published (apart from the website) through the Microsec e-Szigno signature creation and validation application, and were (may have been) also published in the trusted certificate stores of other software. The CA certificates were primarily intended for electronic signatures. It is necessary to be able to validate digital signatures chaining up to these CAs in the long term (50 years). Since electronic signature (end user) certificates only contain the Subject DN of the issuer CA, the reissued (doppelganger) subordinate CA certificates with the same Subject DN and key are equally suitable for certificate chain building. As a consequence, revocation of some subordinate CA certificates might cause validation errors (false negatives) in some signature validation applications, therefore, Microsec maintained all versions of the CA certificates as valid. This solution worked without problems in the creation and validation of signatures, and Microsec has not received any error reports in relation to this throughout the years. Microsec has always published and promoted the use of the latest version of the CA certificates, but could not prevent the use of the earlier versions. 2016-11 Having set up the CCADB, Mozilla introduced more and more stringent checks, and earlier versions of some of the CA certificates were added to the database. Upon the notification from Mozilla, Microsec investigated the situation, and requested to maintain the validity of the affected CA certificates due to reasons explained above. Mozilla temporarily approved to keep all CA certificate versions valid. 2019-11 In relation to the improvement of the CCADB system Mozilla launched automated tools for audit letter validation (ALV). These controls require each subordinate CA certificate in CCADB to be included in one of the audit letters. The automatic checking is based on the SHA-256 fingerprint of the certificate. 2019-11-22 Microsec was informed about the 9 ALV failures from the Microsec root inclusion bug: https://bugzilla.mozilla.org/show_bug.cgi?id=1445364 Microsec began the investigation of the problem immediately and found the following results. During the validation of the Microsec audit letters the ALV tool found 9 discrepancies, which can be categorized in the following types: 1. Formatting problem in the audit letter (5 out of 9) In the audit letters issued near the beginning of 2019 the SHA-256 fingerprint of some certificates were split in two parts by a page break inside the table cell, so the ALV tool was unable to parse the SHA-256 value. The affected certificates were flagged with ALV failures. Microsec wrote comments to the CCADB to the faults explaining the reason of the fault. Microsec contacted its auditor about this problem. As a result, the new audit letters issued on 2019-12-13 did not have this problem, the ALV tool could parse all certificate fingerprints, so these ALV failures disappeared. 2. Subordinate CA certificates without audit letters (4 out of 9) The earlier versions of the above-mentioned doppelganger subordinate CA certificates were not included in the audit letters (always the latest certificate fingerprint was included), so these (4 cases) were flagged as ALV failures. 2019-11-14 Microsec contacted Mozilla again in this matter, referring to the earlier approval in 2016. Mozilla answered that the approval granted 3 years before could not remain valid indefinitely, so considering the recent changes in the automated checks, it was required that each valid subordinate CA certificate should be covered in an audit letter. Microsec had two options: - revoke the earlier versions of the affected (doppelganger) subordinate CA certificates, or - include them in the scope of the audit and have them appear in the audit letter. As 10 years have elapsed since the deprecation of those certificates and Microsec did not want to have them added to the audit letters, Microsec decided to revoke the certificates in question. 2019-11-29 All earlier versions of the subordinate CA certificates were revoked. The revocation caused errors in the systems of some clients, but all those problems were successfully solved by changing the software configuration. Microsec registered the revocations in the CCADB system, and the corresponding ALV failures disappeared. 3. Doppelganger root certificate 2019-12-12 Microsec contacted Mozilla (Kathleen Wilson) by email to clarify the situation. As a result of the investigation we realized that CCADB contains one of the earlier versions of the "Microsec e-Szigno Root CA 2009" certificate, which is now deprecated and not used. Since root certificates cannot be revoked, following a discussion with Mozilla experts Microsec requested the auditor to include both certificates of the root CA in the audit letter. 2020-02-03 The auditor issued the new attestation which contains both root CA certificates, so the corresponding ALV failures disappeared. As a result, the CCADB displayed no ALV failures from 2020-02-03. Latest actions: 2020-03-17 In order to fully clarify the situation, Microsec made an assessment of all affected certificates. As a result of the in-depth investigation, Microsec determined that 7 of the subordinate certificates, which had been issued from the "Microsec e-Szigno Root CA 2009" root and had already been revoked, were not in the current CCADB registry. It was also found that the unused version of the "Microsec e-Szigno Root CA 2009" Root Certificate of 2009-03-06 was not in the CCADB registry either. 2020-03-24 Microsec uploaded the missing subordinate certificates to the CCADB registry. Microsec also uploaded the "Microsec e-Szigno Root CA 2009" root certificate of 2009-03-06 to the CCADB registry as a doppelganger subordinate certificate. This will result in an ALV failure (reporting lack of certification) as the root certificate cannot be revoked and it is not included in the current audit attestation letters. To resolve this warning, Microsec shall request a new audit report from the auditor that includes this third root certificate too. Until this new audit attestation is uploaded, considering the full transparency provided by the above detailed investigation and submissions, Microsec requests Mozilla to disregard this temporary ALV failure. 3. Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem. A statement that you have will be considered a pledge to the community; a statement that you have not requires an explanation. The issuance of the certificates happened in 2009 and did not affect any end user certificates directly. None of the CA services was stopped but only the latest version of the doppelganger root or subordinate certificates were published in the root stores, the software applications etc. 4. A summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued. The problem affects 2 doppelganger root certificates and 10 doppelganger subordinate certificates. There were no end user certificates affected. 5. The complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem. The problem affects the following certificates: Root certificates: Microsec e-Szigno Root CA 2009 ( 194998 ) Microsec e-Szigno Root CA 2009 ( 2370027485 ) Subordinate certificates: Advanced Class 2 e-Szigno CA 2009 ( 2629646531 ) Advanced Class 2 e-Szigno CA 2009 ( 2629646790 ) Advanced Class 3 e-Szigno CA 2009 ( 12722207 ) Advanced Class 3 e-Szigno CA 2009 ( 12726032 ) Advanced Pseudonymous e-Szigno CA 2009 ( 2629646543 ) Advanced Pseudonymous e-Szigno CA 2009 ( 2629646669 ) Qualified e-Szigno CA 2009 ( 2629646584 ) Qualified e-Szigno CA 2009 ( 26312004 ) Qualified Pseudonymous e-Szigno CA 2009 ( 2629646659 ) Qualified Pseudonymous e-Szigno CA 2009 ( 26312005 ) 6. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now. The existence of the doppelganger CA certificates was known but it did not cause any problems in the operation of the CA nor in the CCADB until it was reported in the form of ALV failures. 7. List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things. Microsec learned from this problem and makes much deeper analysis of all the requirements before issuing a new root or subordinate certificate. Microsec does not plan to issue new subordinate certificates in the RSA hierarchy any more due to the close sunset date of the RSA algorithm with 2048-bit keys. It was the main reason to set up the ECC-based CA hierarchy. Due to the long-term preservation requirement Microsec plans to operate the RSA hierarchy to support CRL and OCSP responses till the expiry date of the root and subordinate certificates. Presently Microsec has one ALV failure in the CCADB regarding the earliest version of the "Microsec e-Szigno Root CA 2009" Root Certificate. To resolve this warning, Microsec will request a new audit report from the auditor that includes this third root certificate too. There is no deadline for this but Microsec will inform Mozilla when the date is agreed by the auditor. Until this new audit attestation is uploaded, considering the full transparency provided by the above detailed investigation and submissions, Microsec requests Mozilla to disregard this temporary ALV failure. _______________________________________________ dev-security-policy mailing list [email protected] https://lists.mozilla.org/listinfo/dev-security-policy
