I'm not sure this moves us any closer to addressing https://github.com/mozilla/pkipolicy/issues/155 , but again, I could be misunderstanding the goals here.
This language doesn't seem to quite address the situation raised in https://github.com/mozilla/pkipolicy/issues/155#issue-359674220 I tried to capture in my previous reply what I understood the goals to be, but I may not have done it clearly, so here's me trying to show my work. My understanding is the goals are: 1. Make it clear what CAs are supposed to do if they are informed of qualifications or findings, as WebTrust does. 2. Make it clear what CAs are supposed to do if they are informed of non-conformities (major or minor), as ETSI does. For both of these (which are just two ways of saying the same thing), the expectation is that the CA will file an incident report. As called out in https://wiki.mozilla.org/CA/Responding_To_An_Incident , Mozilla already expects a plan of milestones for remediation, so this is an existing MUST level requirement. Further, in these situations, the proposal was to enumerate some of the additional steps that may be taken, as have been taken in the past, to make sure the expectations are clear. 3. In some situations, Mozilla may require that the CA provide period of time audits sooner than may otherwise be expected, which include confirming that the finding, qualification, or non-conformity has been addressed. 4. In some situations, Mozilla may further require regular audits on an increased frequency, to ensure no further non-conformities are introduced, such as during post-remediation actions. That's what I understood the original proposal to be, which all seems to make sense. I agree with you that https://github.com/mozilla/pkipolicy/issues/150 is basically encapsulated within the existing 155 issue. https://github.com/mozilla/pkipolicy/issues/146 is a different issue than a question of qualifications/findings, and is about a rejection of an audit. For example, an audit report may have no findings or qualifications, despite there having been incidents, and Mozilla decides it cannot rely upon the audit on the basis of these facts. That's about clarifying that: 5. Mozilla may, on receipt of an audit report, determine it to be insufficient for the purposes of the Mozilla Root Program. In such circumstances, the CA may be required, at their own expense, to obtain a new audit to cover the same period in time. 6. Further, Mozilla may, at its sole discretion, require that a different auditor be used in order to provide sufficient assurance for the Mozilla user community. This is basically saying that an audit is simply a report printed on a piece of paper, it is not imbued with special properties, and while ideally, everything works, auditors, like CAs, can make mistakes that undermine trust and confidence in the report. In these circumstances, it is the CA's responsibility to remediate that for continued participation. However, I'm not sure where https://github.com/mozilla/pkipolicy/issues/155#issuecomment-1009460324 is coming into play, given these scenarios. It mostly seems to be a combination of: A) Things the BRs already require (bullet 1, bullet 2) B) Things already covered by existing expectations (bullet 4, 6) C) Things already mentioned in policy (bullet 3, 9, 10 = Section 7.3) That said, D introduces new things (bullet 5 and bullet 8, PIT and POT, even though 155 originally discouraged PIT) When we get to the wording below, we've got a somewhat ambiguous and watered bullet 4 in the first 1.5 sentences, then bullets 5/8 without detail, and then the items from C. What this misses, though, is the original goal of making it clear "We expect an incident report (and as soon as you know - i.e. don't wait until the public report if you already know about an issue)", and then also making it clear that, for some incidents, additional audits may be required (#3 and #4), ideally with details, and ideally, only POT. We're still missing language for #5/#6, which tie back to #3/#4 - namely, that there are circumstances that Mozilla may require additional audits, and potentially at increased frequency. It's not that I see harm in referencing Section 7.3, but it seems unnecessary, because that part is unambiguous in 7.3. What isn't as clear, even though it's very much a past event, is #3, #4, #5, and #6 are possible outcomes, and that #1 and #2 are expected. Tying this all together now: To Section 2.4, the proposed new language becomes: - Any matters discovered during the course of an audit, such as a qualification, a modified opinion, or a non-conformity, are considered incidents, and should be reported as soon as the CA is made aware. In some cases, there may be disagreement between the CA and the auditor on the nature or severity of the situation. CAs MUST (should?) also report these as incidents to Mozilla. In these circumstances, they may be resolved as INVALID if Mozilla determines this to be compliant or not a matter of concern. The edits to 2.5 are less clear, because broadly, #3, #4, #5, #6 (in my list above) are all tied to audits. Perhaps a new 3.3, making the current 3.3 a 3.4, and """ ### 3.3 Additional Audits ### In response to incidents (see Section 2.4), Mozilla MAY require the CA operator to submit one or more additional audits to provide sufficient assurance that the incident has been remediated. Such audits will be expected sooner than the CAs next scheduled audit, and thus may be expected to be for a period less than a year. Mozilla MAY, at its sole discretion, determine that an audit provided does not provide sufficient assurance that the requirements within this section have been sufficiently met. In such circumstances, CAs will be expected to obtain a new audit, at the CA's expense, for the period of time in question. Additionally, depending on the nature of concerns with the audit, Mozilla MAY require the CA obtain such an audit from a new auditor. """ This seems to capture the original issues, and I think, when combined with Section 7.3, covers all the scenarios discussed above? Basically, I think the shift needs to be from thinking about "Sanctions for incidents" (which 7.3 mostly covers), and instead being clear that "Sometimes, additional audits are needed". I grouped these two cases (audits for remediation and audits because crappy auditors) into 3.3, because it made most sense to me, but I can also understand an argument for splitting it out into a 2.5 & 3.3. It just felt a little weird to get too detailed about audits in 2.5 before Section 3 had been introduced, which is why I propose a 3.3 On Wed, Jan 26, 2022 at 5:58 PM Ben Wilson <[email protected]> wrote: > Here is another re-phrasing, > > "Mozilla expects the timely remediation of the problems that caused or > gave rise to the incident. Mozilla MAY require the CA operator to submit a > plan of action with milestones or additional audits to ensure remediation > and to regain confidence in the CA operator. Reoccurring incidents with the > same underlying cause, or failure to remediate the causes giving rise to > incidents, will lead to sanctions. Mozilla MAY impose sanctions, including > but not limited to the full or partial disablement of certificates as set > forth in section 7.3 by adding certificates to OneCRL, removing trust bits > from root certificates, or removing root certificates from the trust > store." > > On Wed, Jan 26, 2022 at 12:37 PM Ryan Sleevi <[email protected]> wrote: > >> >> >> On Wed, Jan 26, 2022 at 1:26 PM Ben Wilson <[email protected]> wrote: >> >>> As my comment on the issue at the time tried to capture, the scenario on >>>> the bug was about the expectations for CAs when they receive qualified >>>> audits, and how that may be treated. >>>> >>> >>> I believed that in response to this, we need to make clearer that >>> qualifications in audits are "incidents," which need to be tracked in >>> Bugzilla through to remediation. I think that many CAs have understood that >>> this has already been the practice for several years, as evidenced by the >>> incidents filed by CAs in Bugzilla, but again, we needed to make sure that >>> it was stated in policy. >>> >> >> Yup, I think this is a great clarification to make, namely: >> >> Whenever a CA becomes aware that they have failed to meet requirements, >> such as via an external report from end-users, an internal discovery, or >> feedback during an audit or consultation, the CA should file a CA incident >> report to disclose that to the community. Additionally, CAs should consider >> raising issues that are flagged by auditors, but which may ultimately be >> determined not to be issues, both to provide an opportunity to ensure no >> misunderstandings, and to provide this as feedback and datapoints for how >> requirements can be clarified or improved in the future. These can be filed >> as Incident Reports, which can always be closed as Resolved/Invalid if the >> conclusion is supported, or raised on m.d.s.p. as items for discussion. >> >> I think DigiCert has been great about that last point, and I think a few >> other CAs have as well. I do know some CAs raise questions through their >> auditors to the [email protected] list, which offers yet another >> venue. While I think (incident or m.d.s.p.) is best, it seems like these >> would all be valuable clarifications, especially since the goal is "better, >> together" >> >> >>> Is there any circumstance where any of the MAYs not already MUSTs in the >>>> BRs wouldn’t be appropriate? Is it necessary to even enumerate the actions >>>> Mozilla may take, for those that are optional, versus addressing them as >>>> they arise (e.g. in incident reports)? >>>> >>> >>> I'm inviting comments and suggestions on improving the language. >>> >> >> Gotcha. I think that the following part: >> Mozilla MAY require revocation of those leaf certificates or intermediate >> certificates that suffered or that are considered defective because of the >> incident. Mozilla expects the timely remediation of the problems that >> caused or gave rise to the incident. Mozilla MAY require the CA operator to >> submit a plan of action with milestones or additional audits to ensure >> remediation and to regain confidence in the CA operator. >> >> Could potentially be reworded as: >> >> Mozilla expects the timely remediation of the problems that caused or >> gave rise to the incident, such as defined by the Baseline Requirements. >> Mozilla expects the CA operator to submit a plan of action with milestones >> and MAY require additional audits to ensure remediation and to regain >> confidence in the CA operator. >> >> Basically, this shifts the revocation expectation out - because that's >> covered by the BRs or (eventually) the S/MIME BRs, makes it clear that the >> expectation of actionable timeline (which is part of the incident report >> template) isn't conditional, but that there MAY be additional requirements >> beyond the resolutions set forth by the BRs. Then you can keep the rest of >> the proposed paragraph, which defines steps that Mozilla may take >> above-and-beyond the BRs. >> >> WDYT? >> > -- You received this message because you are subscribed to the Google Groups "[email protected]" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/a/mozilla.org/d/msgid/dev-security-policy/CAErg%3DHEOoN81cYr3%2B97t6XDLwFfTMDF8X6AZYhVAqVtxK2_6Fw%40mail.gmail.com.
