Dear Ben, Kathleen, Mozilla Security Policy Team & Community,
We write to express our concern over the proposed decision to remove E-Tuğra Root CA Certificates from Mozilla's Root Store. We have been serving as a trusted certificate authority for the past 15 years, managing not only SSL certificates but also providing qualified signature services. Our rich history and broad experience in the field have allowed us to continuously improve our security measures and maintain a strong commitment to the standards of our industry. We acknowledge the seriousness of the situation and respect the Mozilla and community's commitment of maintaining the utmost security standards, but we would like to highlight the following points, which we believe are pivotal to the discussion: 1. Immediate Action: As soon as we were alerted about the potential breach, we took immediate action to assess its impact on our certificate authority (CA) system. Our thorough examination confirmed that no certificate was affected as a result of the incident. 2. Timely Disclosure: We notified Mozilla ( https://bugzilla.mozilla.org/show_bug.cgi?id=1801345) and the broader community about the incident promptly, in line with our commitment to transparency. We opened the discussion on the Mozilla community forum to ensure that all stakeholders were informed and could contribute to the resolution. 3. Proactive Response: To ensure that our systems were safe, we conducted penetration testing and completed it successfully. The results, which we shared with the community, validated our initial assessment: no certificates were compromised in any way. 4. Ian Caroll's Assumptions: We understand Mr. Ian Caroll's concerns. However, his assumptions that there must have been an impact on our certificates are not substantiated by the facts. We noticed multiple attempts in our logs where Ian Caroll tried to prove his hypothesis. However, no certificate breach was observed, reaffirming our assessment. 5. Language Barriers: We apologize if our responses have not been as prompt or satisfying as expected. We assure you that our priority has always been addressing this issue with complete dedication, but language barriers may have slowed our communication at times. 6. Third-Party Verification: We are in the process of providing all relevant information related to the incident to our auditors for independent verification. This is part of our ongoing commitment to maintaining transparency and trust with the community. Therefore, we respectfully disagree with the proposed decision to remove E-Tuğra's Root CA certificates from Mozilla's Root Store, as it appears to be based on assumptions rather than confirmed impacts. We kindly ask the community to consider our points above and wait for the independent audit results, which we believe will support our assessment. We respect the responsibilities delegated to Mozilla as a custodian of public trust and we will continue to work diligently to rebuild confidence in our CA operations. We appreciate your understanding and your continued trust in E-Tuğra. Thanks On Wednesday, 12 July 2023 at 02:29:16 UTC+5 Ben Wilson wrote: > All, > > > Thanks again to all who contributed your expertise and resources in > investigating and commenting on the e-Tugra breach, as described in a blog > post by Ian Carroll <https://ian.sh/etugra> and in subsequent discussions > here on the Mozilla dev-security-policy list > <https://groups.google.com/a/mozilla.org/g/dev-security-policy/c/yqALPG5PC4s/m/sIkv6eLAAAAJ>, > > and on the CCADB Public list > <https://groups.google.com/a/mozilla.org/g/dev-security-policy/c/yqALPG5PC4s/m/sIkv6eLAAAAJ>, > > and in Bugzilla <https://bugzilla.mozilla.org/show_bug.cgi?id=1801345>. > > Summary > > We have determined that the combination of concerns about e-Tugra, which we > previously summarized > <https://groups.google.com/a/mozilla.org/g/dev-security-policy/c/C-HrP1SEq1A/m/b5ccAO5fAwAJ>, > > indicates that the risk of keeping the current e-Tugra root certificates in > Mozilla’s root store outweighs the benefits to our users. Adequate > change-control processes were not in place, and non-CA systems with > connections to CA systems were not being adequately protected. The security > holes that were initially reported were concerning in themselves, and alone > they might not have led us to our decision, but these were compounded by > e-Tugra’s inadequate responses, as recorded in Bugzilla > <https://bugzilla.mozilla.org/show_bug.cgi?id=1801345>. > > Incident Response > > As stated on the Maintenance and Enforcement > <https://wiki.mozilla.org/CA/Maintenance_and_Enforcement#Potential_Problems.2C_Prevention.2C_Response> > > wiki page, we want to see that: > > > > > - > > A CA takes responsibility for their own actions. > - > > Incidents are taken with an appropriate level of seriousness. > - > > Incidents are handled with urgency. > - > > Root cause analysis is performed. > - > > Any questions posed, by anyone, are answered quickly and in detail. > > > - > > A reasonably-detailed incident report > <https://wiki.mozilla.org/CA/Responding_To_An_Incident#Incident_Report> > is made public on what happened, why, and how things have changed to make > sure it won’t happen again. > > e-Tugra’s responses fell short of these expectations. > > Incident Report > > Our guidance on incident reports > <https://wiki.mozilla.org/CA/Responding_To_An_Incident#Incident_Report> > (that has recently moved to the CCADB website > <https://www.ccadb.org/cas/incident-report#incident-reports>) states: > “The incident report should explain how systems or processes failed, how > the mis-issuance or incident was made possible, and why the problem was not > detected earlier. In addition to the timeline of responding to and > resolving the incident, the incident report should explain how the CA > Owner’s systems or processes will be made more robust, and how other CAs > may learn from the incident.” > > e-Tugra’s incident report ( > https://bugzilla.mozilla.org/attachment.cgi?id=9330825) was not > reasonably detailed, downplayed the seriousness of the incident, and did > not provide details on remediation steps. For instance, we requested a > timeline of the events that occurred before November 2022, but e-Tugra only > responded with “Oct 10, 2022: An Internal application was published in > Internet Zone.” The incident report did not contain a comprehensive root > cause analysis, and e-Tugra did not explain how things have changed to > ensure incidents like these won’t happen again. We expected more detailed > information and a stronger sense of responsibility from e-Tugra. > > Urgency > > e-Tugra’s slow and inconsistent responses during the course of the > investigation did not portray appropriate urgency. e-Tugra’s > representatives initially said that there were processes in place to > conduct penetration testing and vulnerability assessments of its systems. > They said that penetration testing would start on November 18, 2022. They > also said that information about the penetration test would be shared as > soon as it became available. On December 12, we requested that information > be provided by January 6, 2023, but it wasn’t until February 2, 2023, > (after prompting) that e-Tugra admitted that they were still getting > started with the penetration testing company. On February 28 we reminded > e-Tugra to provide weekly updates. > > The first information on penetration testing was not made available until > April 2, 2023, and then the executive summary provided only a few brief > statements. The executive summary from a follow-up report also lacked > sufficient detail. e-Tugra indicates that everything will be provided to > its auditors during its next audit. However, this highlights not only the > lack of urgency with which this has been viewed by e-Tugra, but also the > lack of information that we have to objectively assess whether meaningful > change has taken or will take place at e-Tugra. > > Proficiency > > e-Tugra’s statements during the course of the investigation were > inconsistent and in some cases contradictory. e-Tugra representatives > downplayed the severity of the incident by responding, “Only three > documents related to some users' (non SSL) agreements are accessed by the > reporter” and “No sensitive information disclosure revealed in this issue”. > They also said, “no sensitive information is logged in the logging system”. > However, these statements were refuted by the fact that the contents of > 568,647 emails and 251,230 documents were accessible, including access to > password resets and confirmation codes. > > e-Tugra representatives first said there was “no connection” with CA > systems and then said that there was a "one-way flow". Then they said that > their customer portal would allow reissuance for already-validated domain > validation certificates, but that the accessed application had no impact on > the process. This was contradictory because the password-reset function > could have been compromised with information in the log to change a user’s > password. > > Root Removal > > The responses from e-Tugra’s representatives demonstrate that this CA > lacks the expertise needed to operate a sufficiently secure environment for > a publicly trusted CA. As such, we will remove the following root > certificates in our next batch of changes to Mozilla’s root store: > > E-Tugra Global Root CA RSA v3 > > SHA-1 Fingerprint: E9A85D2214521C5BAA0AB4BE246A238AC9BAE2A9 > > SHA-256 Fingerprint: > EF66B0B10A3CDB9F2E3648C76BD2AF18EAD2BFE6F117655E28C4060DA1A3F4C2 > > E-Tugra Global Root CA ECC v3 > > SHA-1 Fingerprint: 8A2FAF5753B1B0E6A104EC5B6A69716DF61CE284 > > SHA-256 Fingerprint: > 873F4685FA7F563625252E6D36BCD7F16FC24951F264E47E1B954F4908CDCA13 > > Note: We will also be removing the expired “e-Tugra Certification > Authority” root certificate, as previously planned. > > Follow-up > > e-Tugra may continue to be a reseller to another CA operator with root > certificates in Mozilla’s root store, as long as that other CA operator > ensures correct systems, procedures, and processes in relation to reselling > certificates within their CA hierarchy, and all domain validation and > certificate issuance is performed by that other CA whose root certificates > are included in Mozilla’s root store. > > e-Tugra may not become an externally-operated subordinate CA > <https://www.mozilla.org/en-US/about/governance/policies/security-group/certs/policy#84-externally-operated-subordinate-cas> > > chaining up to a root certificate in Mozilla’s root store, and in order for > e-Tugra representatives to apply to have new root certificates included in > Mozilla’s root store in the future, they would need to: > > - > > Provide sufficiently detailed artifacts and testing to demonstrate > that they have implemented an infrastructure with processes in place to > ensure that systems are rigorously secured and thoroughly hardened. > - > > Create new root certificates within the properly configured/secured > infrastructure. > - > > Re-apply for inclusion in Mozilla’s root store, and go through Mozilla’s > full root inclusion process > <https://wiki.mozilla.org/CA/Application_Process>. > > > Thanks, > > Ben and Kathleen > > On Mon, Jun 5, 2023 at 11:36 AM Ben Wilson <[email protected]> wrote: > >> Dear Mozilla Community, >> >> This email relates to the e-Tugra breach that was described in a blog >> post by Ian Carroll <https://ian.sh/etugra> and subsequent discussions >> here >> <https://groups.google.com/a/mozilla.org/g/dev-security-policy/c/yqALPG5PC4s/m/sIkv6eLAAAAJ> >> >> and in CCADB Public >> <https://groups.google.com/a/mozilla.org/g/dev-security-policy/c/yqALPG5PC4s/m/sIkv6eLAAAAJ> >> >> and Bugzilla <https://bugzilla.mozilla.org/show_bug.cgi?id=1801345>. We >> are grateful for the involvement of various individuals and >> organizations, particularly Ian Carroll and Google Chrome, who have >> contributed their expertise and resources while investigating this breach. >> >> We are now opening this discussion to help determine whether the e-Tugra >> root certificates should be removed from Mozilla’s Root Store. We will >> greatly appreciate your thoughtful and constructive feedback on this. >> >> Below are some questions for us to consider in this discussion. >> >> What were the main concerns raised by the community during the >> discussions that took place in Bugzilla Bug #1801345 >> <https://bugzilla.mozilla.org/show_bug.cgi?id=1801345>? (this is not a >> complete list; details may be found in the bug) >> >> - >> >> Mr. Carroll indicated that he was able to log in and conduct >> reconnaissance on e-Tugra’s email and document storage systems, gaining >> access to customer PII. >> - >> >> There were security holes in e-Tugra’s internal systems that >> existed because access to internal resources was not adequately >> secured, >> namely, default passwords on some administrative tools had not been >> changed, and internal applications were left exposed without >> appropriate >> access control mechanisms. >> - >> >> Statements by e-Tugra about the lack of impact were refuted by the >> fact that the contents of 568,647 emails and 251,230 documents were >> accessible, including access to password resets and confirmation codes. >> - >> >> e-Tugra indicated in their incident report that they would conduct >> additional vulnerability scanning and penetration testing before the end >> of >> 2022. They also said that they were preparing a detailed report of the >> problem and would provide more information. >> - >> >> None of this information was provided until April 2023. >> - >> >> The explanations in subsequent reports and executive summaries of >> penetration testing did not fully answer the questions that had been >> asked. >> - >> >> A detailed analysis was not provided, no root cause analysis was ever >> provided, and the output of the penetration test (i.e., reports) did not >> offer any detail addressing concerns raised in the incident report. (The >> penetration test reports only provided brief statements and insufficient >> information to evaluate the security posture of e-Tugra.) >> >> Does the benefit of having e-Tugra root certificates in Mozilla’s Root >> Store outweigh the risk? >> >> - >> >> The breach that was found by Mr. Carroll may indicate that the CA >> operators may not have the level of expertise necessary to securely >> operate >> a publicly trusted CA. >> - >> >> The slow response by e-Tugra may also be an indication that they may >> not be prepared to respond to concerns and incidents in the required >> timely >> manner. >> >> How would the distrust of e-Tugra affect our users? >> >> - >> >> Since the e-Tugra root certificates are not currently being used, >> there would not be any impact to our users. >> - >> >> e-Tugra is not currently issuing TLS certificates within their own CA >> hierarchy. Instead, e-Tugra has been acting as a reseller for the SSL.com >> CA, which means that all domain validation and certificate issuance is >> being performed by systems operated by the SSL.com CA, so SSL.com is the >> party currently responsible for issuance/security procedures. >> >> If distrusted, could e-Tugra continue to be a reseller for a publicly >> trusted CA? >> >> - >> >> Yes, as long as the publicly trusted CA ensures correct >> systems/procedures/processes in relation to reselling certificates within >> their CA hierarchy. >> >> If distrusted, could e-Tugra re-apply to have their root certificates >> included in Mozilla’s Root Store? >> >> - >> >> If we decide to remove the current e-Tugra roots, then in order for >> e-Tugra to apply to have their root certificates included in Mozilla’s >> Root >> Store again, they would need to: >> - >> >> Implement an infrastructure that is properly secure and tested, >> with sufficiently detailed artifacts to show that the systems are >> properly >> configured. >> - >> >> Then create new root certificates within the properly secured >> infrastructure. >> - >> >> Then re-apply for inclusion in Mozilla’s root store. >> - >> >> As always, the onus is on the CA operator to demonstrate their >> ability to operate a publicly trusted CA hierarchy. >> >> How are other root stores addressing this issue? >> >> - >> >> Google Chrome announced >> >> <https://groups.google.com/a/ccadb.org/g/public/c/SXAeHT04TFc/m/TIh1ANmHAwAJ> >> >> that they will remove the “E-Tugra Global Root CA ECC v3” and “E-Tugra >> Global Root CA RSA v3” roots from the Chrome Root Store. >> - >> >> Apple stated >> >> <https://groups.google.com/a/ccadb.org/g/public/c/SXAeHT04TFc/m/B_snEIWZAwAJ> >> >> that the e-Tugra roots are not in the Apple Root Store. >> >> Given that SSL.com has been the operator of the issuing CA, e-Tugra’s >> breach has not necessitated that we take any urgent safeguarding actions. >> However, e-Tugra’s problems give us reason to question whether the e-Tugra >> CA should continue to be in our root store, based on the level of expertise >> required to securely operate a publicly trusted CA infrastructure. >> Additionally, e-Tugra’s inadequate responses to the problem and resulting >> requests for information, give us reason to question e-Tugra’s commitment >> to the important role that CAs have in ensuring security on the web. >> >> We appreciate your engagement and dedication, and we kindly request that >> all participants adhere to our Community Participation Guidelines >> <https://www.mozilla.org/en-US/about/governance/policies/participation/>, >> fostering a respectful and constructive environment for dialogue. >> >> Thanks, >> >> Ben and Kathleen >> >> PS: For your reference, below are excerpts from >> https://wiki.mozilla.org/CA/Maintenance_and_Enforcement >> >> CA behavior >> >> - >> >> Did the CA notice the error and take appropriate action in a timely >> manner? >> - >> >> Was the threat or error properly contained? >> - >> >> Did the CA notice the hacker intrusion and take appropriate action in >> a timely manner? >> - >> >> Were the CA's defense mechanisms current and sufficient? >> - >> >> Was the intrusion appropriately contained? >> - >> >> Was the CA's behavior reckless? >> - >> >> Did the CA try to cover up the mistake/threat rather than follow >> Mozilla's Security Policy? >> >> In incident response, Mozilla is looking for the following factors: >> >> - >> >> A CA takes responsibility for their own actions. >> - >> >> Incidents are taken with an appropriate level of seriousness. >> - >> >> Incidents are handled with urgency. >> - >> >> Root cause analysis is performed. >> - >> >> Any questions posed, by anyone, are answered quickly and in detail. >> - >> >> A reasonably-detailed incident report >> <https://wiki.mozilla.org/CA/Responding_To_An_Incident#Incident_Report> >> is made public on what happened, why, and how things have changed to make >> sure it won’t happen again >> >> >> -- You received this message because you are subscribed to the Google Groups "[email protected]" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/a/mozilla.org/d/msgid/dev-security-policy/9a6c49f0-9bdf-4775-87c9-19c422466b9fn%40mozilla.org.
