On Tue, May 19, 2020 at 2:22 PM Matthias van de Meent <matthias.vandeme...@cofano.nl> wrote: > I agree that for any one bug, this metadata is not anything to make > decisions over, but when looking over e.g. the last 3 years, you can > start making more informed guesses on the metadata only. E.g. when you > find that a CA has consistently had 4-8 compliance issues open with > only sporadic updates, although this doesn't tell anything about this > CA in and of itself, it does give a (basic) sense of concern. But, as > I've heard, the metadata is even less precise than I'd previously > expected, and thus this road to knowledge is yet to be built.
Exactly, for better or worse. I'm not sure, even in our idealized world, we'd necessarily /want/ Bugzilla to be this. Historically, the approach has been to systematize patterns (e.g. https://wiki.mozilla.org/CA:Symantec_Issues , https://wiki.mozilla.org/CA:PROCERT_Issues , https://wiki.mozilla.org/CA:WoSign_Issues ), to try and provide that information distilled in a way that's useful to decision making and policy stakeholders. It's difficult to see that purely encapsulated in the bug metadata, because there's issues where something seemingly small can be quite eye opening, and something seemingly major can be actually quite benign. The dimension of severity is subjective across a number of dimensions, and so there's no quick and pert way of summarizing that. I'm not trying to defend the status quo, to be sure. There's a lot I would love to see improved here, it's just not the highest priority in the overall set of issues. For example, I'd rather see things like ALV for intermediates, improvements to auditor qualifications, improvements to audits themselves, structural improvements to the BRs (for which I have many in-flight pull requests), etc. Similarly, I'm concerned about patterns of CAs simply not responding in a timely fashion ( e.g. https://bugzilla.mozilla.org/show_bug.cgi?id=1563579 ), and would love to improve the tooling to detect and alert on this. Or, for that matter, better tooling to help us digest and correlate responses across CAs, or identify when an issue is a duplicate (e.g. https://bugzilla.mozilla.org/show_bug.cgi?id=1638898 ) or not yet revoked (e.g. http://misissued.com ). I have a long list of things that could be improved, for sure, but alas, time and resource limited :) > To continue on your metaphor: I do not expect Tier 1, but maybe more > like somewhere between Tier 2 en Tier 3? A biweekly, monthly or any > regular check for open compliance issues waiting for a reply from > Mozilla would already improve the (observed) situation tremendously. I don't disagree on room for improvements, for sure, but I'm still going to push back on a Mozilla obligation here, because I don't think that's reasonable. The interesting thing about this particular situation is that if CAs consistently adhered to their existing expectations, it does tend to end up resolved quicker. The ball is placed consistently in the CA's court to drive this to resolution, and they're expected to be continually doing that. This is, in part, due to how Bugzilla searching, filtering, and alerting works. This again goes back to where there's room for improvement here, if you're passionate, and help is welcome :) This goes back to workflow. If CAs followed the defined workflow, "it works". You're right that CAs aren't following the defined workflow, and so it's not working as ideally as possible, and that makes some of the data you want more difficult. Should Mozilla start removing CAs that don't adhere to that work flow? In some extreme cases (e.g. see above), it's probably justified, if not outright necessary. In other cases, that may be seen as extreme. The important thing here, for setting expectations, is that the CA is expected to be continually driving things. That they aren't is an issue, and it's one I'm fully supportive of filing incidents against CAs to improve. _______________________________________________ dev-security-policy mailing list dev-security-policy@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-security-policy
