E-Tugra posted a decent reply to the incident report at https://bugzilla.mozilla.org/show_bug.cgi?id=1801345. I don't have a Bugzilla account and think it would be more appropriate to respond to it here. In general, the timeline they have shared is inspiring, and it is good that they took swift action on my report. They didn't really communicate with me about it during the incident, but it is good to see that it was taken seriously internally.
However, I am still concerned with what E-Tugra considers logical separation from their CA infrastructure. The administrative tools were clearly connected to EJBCA, as the logging tool had explicit references to calling the EJBCA APIs. Additionally, the E-Tugra panel at panel.e-tugra.com.tr had its authentication system compromised due to the ability to see all emails sent from this. How does this website to buy certificates, used by many customers, have logical separation from the CA system? There are many statements about M of N, HSM access, etc which do not appear to be relevant to this issue. Unless E-Tugra is hand-copying purchased certificates into an air-gapped issuance infrastructure, the site to buy certificates will have some degree of connection to the issuance infrastructure. It would be useful to enumerate the specific details here instead of making blanket assertions about logical separation which appear to be wrong. On Tuesday, November 22, 2022 at 9:24:20 AM UTC-8 [email protected] wrote: > Two simple questions that desperately need answers: > > 1) Does E-Tugra have specific controls in place to ensure systems deployed > to production are secured (e.g. default accounts disabled or changed)? > 2) If #1 is sufficient, how and why did they fail? If #1 is not sufficient > why is it missing such fundamental controls? > 3) Has E-Tugra validated ALL other existing servers/services (both public > facing and internal facing) to ensure similar lapses haven't occurred? > > On Tue, Nov 22, 2022 at 7:18 AM Israr Ahmed <[email protected]> wrote: > >> Hi Ian Carroll, >> >> We are working with several internal teams to get their feedback and >> analysis on the reported problems. >> >> We are compiling a detailed report based on feedback from each team >> before we publish it. We hope that we will able to complete this activity >> within a week time. >> >> On Saturday, 19 November 2022 at 00:06:33 UTC+5 [email protected] wrote: >> >>> In the E-Tugra incident report, they state: >>> >>> > Only three documents related to some users' (non SSL) agreements are >>> accessed by the reporter. >>> >>> I just want to clarify that this issue impacted 251,230 uploaded user >>> documents, as shown in their administrative panel >>> <https://imgur.com/a/w94u9lf>, as well as over 500,000 sent emails. I >>> may have only accessed three documents during my testing, but the scope of >>> this issue was much larger. >>> >>> The incident report states: >>> >>> > Customers sensitive information e.g. credentials and similar >>> information could not be obtained and are in safe condition. >>> >>> However, E-Tugra password resets, verification codes, and login codes >>> for SMS were all visible in this portal. Explicit passwords may not have >>> been visible, but many E-Tugra users likely normally log in via SMS, which >>> were impacted by this. Additionally, there were customer portal issues not >>> disclosed in my post and I am unsure if these have been investigated for >>> abuse. >>> >>> The incident report does not show a sufficient level of investigation >>> (or just does not detail enough of it), in my opinion. There's no >>> information on when this issue was introduced and how long they may have >>> been exposed to this issue. There is a statement that activities were >>> reviewed over the past year, but were the issues present for longer than >>> this? A logging system was one of the two impacted systems as well; could >>> logs for this have been tampered with/modified? The customer portal is also >>> not addressed. >>> >>> On Fri, Nov 18, 2022 at 9:10 AM Ryan Hurst <[email protected]> wrote: >>> >>>> In my personal capacity, as I review the published facts that exist >>>> here, there are a few broad things from a BR perspective that stand out to >>>> me: >>>> >>>> - >>>> >>>> All CAs are required to have incident response and compromise >>>> handling procedures. The timeline presented in Ian’s post suggests >>>> that no such plan exists, the plan that exists is insufficient, or the >>>> plan >>>> is not staffed. >>>> >>>> >>>> - >>>> >>>> All CAs are required to perform an annual Risk Assessment. The >>>> existence of administrative consoles on the internet, default >>>> administrative passwords being in use, and the ability to in an >>>> uncontrolled fashion add new administrative users tells us this either >>>> is >>>> not happening or is being performed by individuals with zero >>>> understanding >>>> of security. >>>> >>>> >>>> - >>>> >>>> All CAs are required to perform an annual penetration test. The >>>> existence of these trivial issues suggests that either this was not >>>> happening or it was being performed by individuals with zero >>>> understanding >>>> of security. >>>> >>>> While PKI-related auditors tend to have a limited understanding of PKI >>>> and security, in all my time in this industry they have always >>>> demonstrated >>>> an understanding of IT accounting principles that would be sufficient to >>>> identify these trivial deficiencies. >>>> >>>> I would go so far as to say that these are such basic issues the fact >>>> that e-Turga has continually been able to acquire a sufficiently clean >>>> audit report to stay within the various programs suggests either a >>>> disqualifying skill deficiency or an integrity issue related to the >>>> auditing firms. >>>> >>>> I also think this incident shows that the current root program >>>> requirements and BRs are too generous when it comes to trusting CAs to >>>> implement reasonable incident response procedures. >>>> >>>> Some things that should be considered moving forward include: >>>> >>>> 1. >>>> >>>> Requiring the availability of a documented method for reporting >>>> security concerns. >>>> 2. >>>> >>>> Requiring a minimum response time for the initial response to these >>>> reports. >>>> 3. >>>> >>>> Requiring that the final disposition of these reports be >>>> communicated to the reporter. >>>> 4. >>>> >>>> Requiring that subscribers be notified of any compromises to the >>>> certificate management-related systems. >>>> 5. >>>> >>>> Today only issues related to a security issue requiring revocation >>>> of an intermediate or cert chaining to a trusted root require the >>>> creation >>>> of a bug in Bugzilla, this should be expanded to minimally include >>>> compromise of any certificate management-related system. >>>> 6. >>>> >>>> Requiring that all security incidents be summarized in some >>>> reasonable detail and made available in CCADB for root program >>>> administrators. >>>> 7. >>>> >>>> Requiring the identification of who performed the penetration >>>> testing, a summary of what was assessed, and a summary fo their >>>> findings. >>>> >>>> I also believe a common theme in CA failures like these is the lack of >>>> qualifications and independence of the auditors and security professionals >>>> used to assess the correctness and conformance of the systems. >>>> >>>> I don’t know how best to address this but this incident also seems to >>>> suggest that the requirement to use a “qualified auditor” and penetration >>>> testers with the “skills, tools, proficiency, code of ethics, and >>>> independence” is not sufficient. >>>> >>>> >>>> Ryan Hurst >>>> >>>> (Personal Capacity) >>>> >>>> >>>> >>>> On Thursday, November 17, 2022 at 2:51:29 PM UTC-8 [email protected] wrote: >>>> >>>>> The major issues mentioned in the post are resolved after I notified >>>>> e-Tugra of them. >>>>> >>>>> It is important to note that I did not perform an extensive amount of >>>>> testing of e-Tugra’s systems, given I do not speak Turkish and I found >>>>> enough issues as it is. Until e-Tugra undertakes a comprehensive security >>>>> test of their systems, I would assume there are still other >>>>> vulnerabilities >>>>> present somewhere else. >>>>> >>>>> On Thu, Nov 17, 2022 at 2:42 PM Kurt Seifried <[email protected]> >>>>> wrote: >>>>> >>>>>> >>>>>> >>>>>> On Thu, Nov 17, 2022 at 2:59 PM ke ju <[email protected]> wrote: >>>>>> >>>>>>> I agree that something should be done swiftly. If you did this, who >>>>>>> else could have. Also, has the authority been notified, and have they >>>>>>> simply changed the passwords yet? Ie could a threat actor watching this >>>>>>> list now go there and get into the system after the disclosure >>>>>>> >>>>>> >>>>>> To be clear, I'm not the original reporter. They also posted to the >>>>>> old list which is why I cross-posted it. >>>>>> >>>>>> Their blog entry has a timeline: >>>>>> >>>>>> >>>>>> - Nov 13, 2022 4:10: Initial contact to e-Tugra about >>>>>> administrative systems >>>>>> - Unknown: Administrative systems no longer reachable on the >>>>>> internet >>>>>> - Nov 13, 2022 18:50: Second set of issues reported to e-Tugra, >>>>>> follow-up on initial issues that appeared fixed >>>>>> - Nov 14, 2022 8:35: Initial reply from e-Tugra saying they are >>>>>> working on resolution >>>>>> - Nov 14, 2022 17:18: Asked how to report security issues in the >>>>>> future >>>>>> - Nov 16, 2022 22:52: Notified e-Tugra of impending disclosure >>>>>> - Nov 17, 2022: Disclosed this post >>>>>> >>>>>> >>>>>> >>>>>>> On Thursday, November 17, 2022 at 12:52:37 PM UTC-5 >>>>>>> [email protected] wrote: >>>>>>> >>>>>>>> Normally I would say e-Tugra needs to reissue all their >>>>>>>> certificates or like in this case; it would appear they need to >>>>>>>> reestablish >>>>>>>> that the certificates were issued properly, which means having all >>>>>>>> their >>>>>>>> customers re-create them, establish domain validation, etc. >>>>>>>> >>>>>>>> But in this case, I think it's so severe that they should be >>>>>>>> removed and be made to re-apply to ensure all their security >>>>>>>> controls/processes are up to standard because they clearly are not. >>>>>>>> This >>>>>>>> isn't a little mistake. this shows a massive failure at all levels, >>>>>>>> technically and business-wise (not removing default passwords, >>>>>>>> seriously... >>>>>>>> that's literally the most basic of the basic, what else did they get >>>>>>>> wrong?). >>>>>>>> >>>>>>>> Additionally, did an attacker possibly get a whole set of new >>>>>>>> certificates over the summer for their domain names (just a few days >>>>>>>> ago, etugra renewed everything using their own CA, but they also did a >>>>>>>> lot >>>>>>>> of them in July/August of this year). >>>>>>>> >>>>>>>> https://crt.sh/?q=e-tugra >>>>>>>> https://crt.sh/?q=etugratest.com >>>>>>>> >>>>>>>> Also, should this discussion be moved to >>>>>>>> https://groups.google.com/a/ccadb.org/g/public ? Added >>>>>>>> [email protected]. >>>>>>>> >>>>>>>> On Thu, Nov 17, 2022 at 10:26 AM Ian Carroll <[email protected]> wrote: >>>>>>>> >>>>>>>>> Hi there, >>>>>>>>> >>>>>>>>> Today I published a blog post at https://ian.sh/etugra, >>>>>>>>> describing several serious security issues I discovered in the >>>>>>>>> e-Tugra >>>>>>>>> certificate authority. I was able to obtain access to two e-Tugra >>>>>>>>> administrative systems using default passwords, which disclosed >>>>>>>>> numerous >>>>>>>>> amounts of subscriber PII and verification details, and appeared to >>>>>>>>> impact >>>>>>>>> e-Tugra's domain control validation processes. >>>>>>>>> >>>>>>>>> I am concerned that it is possible for these trivial >>>>>>>>> vulnerabilities to be present in a publicly-trusted certificate >>>>>>>>> authority. >>>>>>>>> In light of the recent Symantec news >>>>>>>>> <https://symantec-enterprise-blogs.security.com/blogs/threat-intelligence/espionage-asia-governments-cert-authority>, >>>>>>>>> >>>>>>>>> certificate authorities are clearly being targeted by nation states, >>>>>>>>> but >>>>>>>>> these vulnerabilities could have been discovered by any amateur >>>>>>>>> security >>>>>>>>> researcher. From what I have seen, I firmly believe that additional >>>>>>>>> security issues likely exist in e-Tugra's infrastructure, and they >>>>>>>>> may >>>>>>>>> already be known to adversaries. >>>>>>>>> >>>>>>>>> The Network and Certificate System Security Requirements require >>>>>>>>> an annual penetration test, or whenever the CA believes there are >>>>>>>>> material >>>>>>>>> changes. Based on this issue, I am concerned that this control is not >>>>>>>>> sufficient to protect certificate authorities against application >>>>>>>>> security >>>>>>>>> issues, and I am concerned that e-Tugra is not following this >>>>>>>>> control. I am >>>>>>>>> also concerned with the lack of vulnerability disclosure programs and >>>>>>>>> bug >>>>>>>>> bounty programs that are operated by CAs in general; indeed no >>>>>>>>> certificate >>>>>>>>> authority at all appears to run a bug bounty program at the moment. >>>>>>>>> >>>>>>>>> I would suggest that e-Tugra be compelled to take remedial actions >>>>>>>>> such as performing a comprehensive penetration test on their external >>>>>>>>> infrastructure, and building processes to ensure that future >>>>>>>>> applications >>>>>>>>> that they deploy are secure. I also believe e-Tugra should ensure >>>>>>>>> that >>>>>>>>> these issues did not have the ability to compromise domain-control >>>>>>>>> validation for any certificates still valid today. >>>>>>>>> >>>>>>>>> -- >>>>>>>>> You received this message because you are subscribed to the Google >>>>>>>>> Groups "[email protected]" group. >>>>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>>>> send an email to [email protected]. >>>>>>>>> To view this discussion on the web visit >>>>>>>>> https://groups.google.com/a/mozilla.org/d/msgid/dev-security-policy/37cff300-b57a-4b38-82e0-a514b4557b07n%40mozilla.org >>>>>>>>> >>>>>>>>> <https://groups.google.com/a/mozilla.org/d/msgid/dev-security-policy/37cff300-b57a-4b38-82e0-a514b4557b07n%40mozilla.org?utm_medium=email&utm_source=footer> >>>>>>>>> . >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> Kurt Seifried (He/Him) >>>>>>>> [email protected] >>>>>>>> >>>>>>> >>>>>> >>>>>> -- >>>>>> Kurt Seifried (He/Him) >>>>>> [email protected] >>>>>> >>>>> -- >>>>> Thanks, >>>>> Ian Carroll >>>>> >>>> >>> >>> -- >>> Thanks, >>> Ian Carroll >>> >> -- >> You received this message because you are subscribed to the Google Groups >> "[email protected]" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> > To view this discussion on the web visit >> https://groups.google.com/a/mozilla.org/d/msgid/dev-security-policy/cc07165b-31bd-4e2d-a96b-e4b6d9611445n%40mozilla.org >> >> <https://groups.google.com/a/mozilla.org/d/msgid/dev-security-policy/cc07165b-31bd-4e2d-a96b-e4b6d9611445n%40mozilla.org?utm_medium=email&utm_source=footer> >> . >> > > > -- > Kurt Seifried (He/Him) > [email protected] > -- You received this message because you are subscribed to the Google Groups "[email protected]" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/a/mozilla.org/d/msgid/dev-security-policy/8ad1e1bf-fa8c-4618-91fa-d3fdce31b3b7n%40mozilla.org.
