Based on conversations with Kathleen, I have simplified the changes made to address these issues. See https://github.com/BenWilson-Mozilla/pkipolicy/commit/7b0cb570fdf5b423b1b267fd01bc2bbb5aced7a9
On Thu, Jan 27, 2022 at 11:22 AM Ben Wilson <[email protected]> wrote: > Hi Ryan, > I have attempted to incorporate your comments and suggestions into a new > draft - > https://github.com/BenWilson-Mozilla/pkipolicy/commit/7c7712467121c92ce25c6211fc19d52a895fc743, > although I moved some suggestions into different sections. > > I inserted the following into 2.4: > > “Any matter discovered during the course of an audit giving rise to a > qualification, a modified opinion, or a non-conformity, is also considered > an incident and MUST be reported to Mozilla as soon as the CA operator is > made aware. In some cases, there may be disagreement between the CA > operator and the auditor on the nature or severity of the situation. > However, CA operators MUST report these as incidents to Mozilla. In these > circumstances, reported incidents may be resolved as INVALID, if Mozilla > determines that there has been no non-compliance or that they are not a > matter of concern.” > > I deleted the section heading for 2.5 (Sanctions) and added to 2.4: > > " Mozilla expects the timely remediation of the problems that caused or > gave rise to the incident. In response to incidents, Mozilla MAY require > the CA operator to submit a plan of action with milestones or to submit one > or more additional audits to provide sufficient assurance that the incident > has been remediated. Such audits will be expected sooner than the CA > operator’s next scheduled audit, and thus may be expected to be for a > period less than a year." > > Added to 3.2 (Auditors): "Mozilla MAY, at its sole discretion, determine > that an audit provided does not provide sufficient assurance that the > requirements within this section 3 have been sufficiently met. In such > circumstances, CA operators will be expected to obtain a new audit, at the > CA operator's expense, for the period of time in question. Additionally, > depending on the nature of concerns with the audit, Mozilla MAY require > that the CA operator obtain such an audit from a new auditor." > > Then, in section 7.3, I added: "Mozilla MAY remove or disable a > certificate if the CA operator has reoccurring incidents with the same > underlying cause or failed to remediate the causes giving rise to incidents. > " > > Ben > > On Wed, Jan 26, 2022 at 5:19 PM Ryan Sleevi <[email protected]> wrote: > >> I'm not sure this moves us any closer to addressing >> https://github.com/mozilla/pkipolicy/issues/155 , but again, I could be >> misunderstanding the goals here. >> >> This language doesn't seem to quite address the situation raised in >> https://github.com/mozilla/pkipolicy/issues/155#issue-359674220 >> >> I tried to capture in my previous reply what I understood the goals to >> be, but I may not have done it clearly, so here's me trying to show my >> work. My understanding is the goals are: >> >> 1. Make it clear what CAs are supposed to do if they are informed >> of qualifications or findings, as WebTrust does. >> 2. Make it clear what CAs are supposed to do if they are informed of >> non-conformities (major or minor), as ETSI does. >> >> For both of these (which are just two ways of saying the same thing), the >> expectation is that the CA will file an incident report. As called out in >> https://wiki.mozilla.org/CA/Responding_To_An_Incident , Mozilla already >> expects a plan of milestones for remediation, so this is an existing MUST >> level requirement. >> >> Further, in these situations, the proposal was to enumerate some of the >> additional steps that may be taken, as have been taken in the past, to make >> sure the expectations are clear. >> >> 3. In some situations, Mozilla may require that the CA provide period of >> time audits sooner than may otherwise be expected, which include confirming >> that the finding, qualification, or non-conformity has been addressed. >> 4. In some situations, Mozilla may further require regular audits on an >> increased frequency, to ensure no further non-conformities are introduced, >> such as during post-remediation actions. >> >> That's what I understood the original proposal to be, which all seems to >> make sense. I agree with you that >> https://github.com/mozilla/pkipolicy/issues/150 is basically >> encapsulated within the existing 155 issue. >> >> https://github.com/mozilla/pkipolicy/issues/146 is a different issue >> than a question of qualifications/findings, and is about a rejection of an >> audit. For example, an audit report may have no findings or qualifications, >> despite there having been incidents, and Mozilla decides it cannot rely >> upon the audit on the basis of these facts. That's about clarifying that: >> >> 5. Mozilla may, on receipt of an audit report, determine it to be >> insufficient for the purposes of the Mozilla Root Program. In such >> circumstances, the CA may be required, at their own expense, to obtain a >> new audit to cover the same period in time. >> 6. Further, Mozilla may, at its sole discretion, require that a different >> auditor be used in order to provide sufficient assurance for the Mozilla >> user community. >> >> This is basically saying that an audit is simply a report printed on a >> piece of paper, it is not imbued with special properties, and while >> ideally, everything works, auditors, like CAs, can make mistakes that >> undermine trust and confidence in the report. In these circumstances, it is >> the CA's responsibility to remediate that for continued participation. >> >> However, I'm not sure where >> https://github.com/mozilla/pkipolicy/issues/155#issuecomment-1009460324 >> is coming into play, given these scenarios. >> >> It mostly seems to be a combination of: >> A) Things the BRs already require (bullet 1, bullet 2) >> B) Things already covered by existing expectations (bullet 4, 6) >> C) Things already mentioned in policy (bullet 3, 9, 10 = Section 7.3) >> >> That said, D introduces new things (bullet 5 and bullet 8, PIT and POT, >> even though 155 originally discouraged PIT) >> >> When we get to the wording below, we've got a somewhat ambiguous and >> watered bullet 4 in the first 1.5 sentences, then bullets 5/8 without >> detail, and then the items from C. What this misses, though, is the >> original goal of making it clear "We expect an incident report (and as soon >> as you know - i.e. don't wait until the public report if you already know >> about an issue)", and then also making it clear that, for some incidents, >> additional audits may be required (#3 and #4), ideally with details, and >> ideally, only POT. >> >> We're still missing language for #5/#6, which tie back to #3/#4 - namely, >> that there are circumstances that Mozilla may require additional audits, >> and potentially at increased frequency. >> >> It's not that I see harm in referencing Section 7.3, but it seems >> unnecessary, because that part is unambiguous in 7.3. What isn't as clear, >> even though it's very much a past event, is #3, #4, #5, and #6 are possible >> outcomes, and that #1 and #2 are expected. >> >> Tying this all together now: >> >> To Section 2.4, the proposed new language becomes: >> - Any matters discovered during the course of an audit, such as a >> qualification, a modified opinion, or a non-conformity, are considered >> incidents, and should be reported as soon as the CA is made aware. In some >> cases, there may be disagreement between the CA and the auditor on the >> nature or severity of the situation. CAs MUST (should?) also report these >> as incidents to Mozilla. In these circumstances, they may be resolved as >> INVALID if Mozilla determines this to be compliant or not a matter of >> concern. >> >> The edits to 2.5 are less clear, because broadly, #3, #4, #5, #6 (in my >> list above) are all tied to audits. Perhaps a new 3.3, making the current >> 3.3 a 3.4, and >> >> """ >> ### 3.3 Additional Audits ### >> >> In response to incidents (see Section 2.4), Mozilla MAY require the CA >> operator to submit one or more additional audits to provide sufficient >> assurance that the incident has been remediated. Such audits will be >> expected sooner than the CAs next scheduled audit, and thus may be expected >> to be for a period less than a year. >> >> Mozilla MAY, at its sole discretion, determine that an audit provided >> does not provide sufficient assurance that the requirements within this >> section have been sufficiently met. In such circumstances, CAs will be >> expected to obtain a new audit, at the CA's expense, for the period of time >> in question. Additionally, depending on the nature of concerns with the >> audit, Mozilla MAY require the CA obtain such an audit from a new auditor. >> """ >> >> This seems to capture the original issues, and I think, when combined >> with Section 7.3, covers all the scenarios discussed above? Basically, I >> think the shift needs to be from thinking about "Sanctions for incidents" >> (which 7.3 mostly covers), and instead being clear that "Sometimes, >> additional audits are needed". I grouped these two cases (audits for >> remediation and audits because crappy auditors) into 3.3, because it made >> most sense to me, but I can also understand an argument for splitting it >> out into a 2.5 & 3.3. It just felt a little weird to get too detailed about >> audits in 2.5 before Section 3 had been introduced, which is why I propose >> a 3.3 >> >> On Wed, Jan 26, 2022 at 5:58 PM Ben Wilson <[email protected]> wrote: >> >>> Here is another re-phrasing, >>> >>> "Mozilla expects the timely remediation of the problems that caused or >>> gave rise to the incident. Mozilla MAY require the CA operator to submit a >>> plan of action with milestones or additional audits to ensure remediation >>> and to regain confidence in the CA operator. Reoccurring incidents with the >>> same underlying cause, or failure to remediate the causes giving rise to >>> incidents, will lead to sanctions. Mozilla MAY impose sanctions, including >>> but not limited to the full or partial disablement of certificates as set >>> forth in section 7.3 by adding certificates to OneCRL, removing trust bits >>> from root certificates, or removing root certificates from the trust >>> store." >>> >>> On Wed, Jan 26, 2022 at 12:37 PM Ryan Sleevi <[email protected]> wrote: >>> >>>> >>>> >>>> On Wed, Jan 26, 2022 at 1:26 PM Ben Wilson <[email protected]> wrote: >>>> >>>>> As my comment on the issue at the time tried to capture, the scenario >>>>>> on the bug was about the expectations for CAs when they receive qualified >>>>>> audits, and how that may be treated. >>>>>> >>>>> >>>>> I believed that in response to this, we need to make clearer that >>>>> qualifications in audits are "incidents," which need to be tracked in >>>>> Bugzilla through to remediation. I think that many CAs have understood >>>>> that >>>>> this has already been the practice for several years, as evidenced by the >>>>> incidents filed by CAs in Bugzilla, but again, we needed to make sure that >>>>> it was stated in policy. >>>>> >>>> >>>> Yup, I think this is a great clarification to make, namely: >>>> >>>> Whenever a CA becomes aware that they have failed to meet requirements, >>>> such as via an external report from end-users, an internal discovery, or >>>> feedback during an audit or consultation, the CA should file a CA incident >>>> report to disclose that to the community. Additionally, CAs should consider >>>> raising issues that are flagged by auditors, but which may ultimately be >>>> determined not to be issues, both to provide an opportunity to ensure no >>>> misunderstandings, and to provide this as feedback and datapoints for how >>>> requirements can be clarified or improved in the future. These can be filed >>>> as Incident Reports, which can always be closed as Resolved/Invalid if the >>>> conclusion is supported, or raised on m.d.s.p. as items for discussion. >>>> >>>> I think DigiCert has been great about that last point, and I think a >>>> few other CAs have as well. I do know some CAs raise questions through >>>> their auditors to the [email protected] list, which offers yet >>>> another venue. While I think (incident or m.d.s.p.) is best, it seems like >>>> these would all be valuable clarifications, especially since the goal is >>>> "better, together" >>>> >>>> >>>>> Is there any circumstance where any of the MAYs not already MUSTs in >>>>>> the BRs wouldn’t be appropriate? Is it necessary to even enumerate the >>>>>> actions Mozilla may take, for those that are optional, versus addressing >>>>>> them as they arise (e.g. in incident reports)? >>>>>> >>>>> >>>>> I'm inviting comments and suggestions on improving the language. >>>>> >>>> >>>> Gotcha. I think that the following part: >>>> Mozilla MAY require revocation of those leaf certificates or >>>> intermediate certificates that suffered or that are considered defective >>>> because of the incident. Mozilla expects the timely remediation of the >>>> problems that caused or gave rise to the incident. Mozilla MAY require the >>>> CA operator to submit a plan of action with milestones or additional audits >>>> to ensure remediation and to regain confidence in the CA operator. >>>> >>>> Could potentially be reworded as: >>>> >>>> Mozilla expects the timely remediation of the problems that caused or >>>> gave rise to the incident, such as defined by the Baseline Requirements. >>>> Mozilla expects the CA operator to submit a plan of action with milestones >>>> and MAY require additional audits to ensure remediation and to regain >>>> confidence in the CA operator. >>>> >>>> Basically, this shifts the revocation expectation out - because that's >>>> covered by the BRs or (eventually) the S/MIME BRs, makes it clear that the >>>> expectation of actionable timeline (which is part of the incident report >>>> template) isn't conditional, but that there MAY be additional requirements >>>> beyond the resolutions set forth by the BRs. Then you can keep the rest of >>>> the proposed paragraph, which defines steps that Mozilla may take >>>> above-and-beyond the BRs. >>>> >>>> WDYT? >>>> >>> -- You received this message because you are subscribed to the Google Groups "[email protected]" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/a/mozilla.org/d/msgid/dev-security-policy/CA%2B1gtaY1rUJrj1sveXZA7_7rpq4kVV3CbEe%3DQrbVMgWsy-c63A%40mail.gmail.com.
