On Tue, 19 May 2020 at 16:22, Ryan Sleevi <[email protected]> wrote: > > On Tue, May 19, 2020 at 5:53 AM Matthias van de Meent > <[email protected]> wrote: >> >> One of the reasons I did this research was to check the track record >> of CAs with regards to compliance and solving compliance issues. As >> you might expect, this is quite difficult when the issues are not >> updated regularly. > > I’m not sure I see the connection. For this scenario, it’s actually necessary > to read the bugs. We do not use the bug metadata (e.g. opened, last updated, > closed date) as part of our own analysis and processing, instead relying on > reading the bugs themselves.
I agree that for any one bug, this metadata is not anything to make decisions over, but when looking over e.g. the last 3 years, you can start making more informed guesses on the metadata only. E.g. when you find that a CA has consistently had 4-8 compliance issues open with only sporadic updates, although this doesn't tell anything about this CA in and of itself, it does give a (basic) sense of concern. But, as I've heard, the metadata is even less precise than I'd previously expected, and thus this road to knowledge is yet to be built. >> The distinction closed / open is, (although skewed) a decent >> indication for a CAs compliance track record and their readiness to >> improve, especially when tracking open issues over time. When the >> issue state is not linked to the actual solving of the compliance >> issue, the skewed indication becomes even worse. > > > Thanks. I don’t think we intend for it to be used as such, and so perhaps > it’s not surprising that you find the current data unsatisfying. I would > disagree that it’s a decent approximation, in part because that’s not > something we guarantee, as you can see. > > I say this not to disagree that it “could” be useful, but it’s considerably > more work and effort, and places the burden back on the Mozilla community to > guarantee, for something that isn’t essential for our efforts or necessary > for our tasks. > > For lack of a better comparison, it is a bit like asking why AIX or BeOS > aren’t a tier 1 configuration ( > https://firefox-source-docs.mozilla.org/build/buildsystem/supported-configurations.html > ). The answer is because it’s significant burden for limited return. Thanks, I had not quite considered this part of the equation. >> The MRSP section 2.4 asks the CA to promptly provide an incident >> report, and regularly update this report until it is closed. My >> opinion is that in this section Mozilla also has an implicit duty to >> the CA - to mark issues as resolved when Mozilla and the CA agree that >> the compliance issue has been resolved. > > > Thanks. I don’t agree with that implicit duty assertion, and I think the only > reason it comes up here is because it’s tied to a use case which is not > presently supported. > > To use the previous metaphor, this is a bit like saying that if AIX/BeOS, > then Mozilla has an implicit duty not to break the build. However, that > ignores the notion of support tiers, and what guarantees, if any, are > provided. I believe this is a bad metaphor for what I understand is Mozillas role in handling Incident Report bugs. MRSP 2.4 specifically calls out that a Mozilla representative must eventually close the bug (or not?), and that until that moment (or forever?) the CA must continue regularly updating this Incident Report. Although I do agree that Mozilla is not responsible for actively driving the Incident Report towards closure (that is as I understand it the role of the CA), I do believe that Mozilla still should take action when the CA is convinced the Incident Report is complete and can be closed or asks for extra guidance - be it resolving the bug, be it responding and asking for missing information. I do believe that a CA has some right to "reasonable" response times when they believe the incident has been handled, just like Mozilla asks for "regular updates" on Incident Reports. Note that I, similar to the MRSP, do not mention what time frames to expect when talking about "reasonable" or "regular" in this context. To continue on your metaphor: I do not expect Tier 1, but maybe more like somewhere between Tier 2 en Tier 3? A biweekly, monthly or any regular check for open compliance issues waiting for a reply from Mozilla would already improve the (observed) situation tremendously. >> Currently, I cannot see the forest for the trees due to so many issues >> waiting to be closed, or having their next-update-by -windows long >> passed, or just plain lack of communication about what is going on. >> This makes it even more difficult to make informed decisions about >> those CAs based on compliance track record. > > > To close this out unambiguously: Using purely the bug metadata (i.e. the > status, last modified date, etc) will not tell you this story. What you are > trying to do is not presently supported. > > Would it be nice? Yes, I have no doubt. But that will require significantly > more tooling and effort, on the part of Mozilla, and significant improvements > by CAs. These are things that can be done, but are not zero cost, and are not > on the most pressing of issues. > > It’s important to remember what this system replaced: threads on this mailing > list. Threads which, by definition, are not “closed”, and which require > reading with and examining in order to extract the timeline and structural > data. The current use of Bugzilla is, for better or worse, a slight > improvement on that system to help distinguish those threads and provide > clearer signals to the CA with fewer messages, but it is not currently a full > replacement for a structural, machine-readable timeline extraction. > > If you’d like to volunteer and contribute to the efforts to improve that, I’m > happy to highlight areas for how additional tooling can help improve. This is > certainly something we’ve discussed as a “nice to have”, but nowhere near the > top of any of our priority lists. Patches to support this workflow would > certainly be welcome. > > Hopefully that helps recalibrate expectations about what is currently > supported, as it sounds like your use case is outside what’s supported. Also, > hopefully it’s clearer that it’s not an unintentional decision, but an > intentional one based on priorities and needs. Much like supporting an > esoteric platform, there are non-trivial initial costs and startup costs, and > as open source maintainers, sometimes the answer needs to be “No” ( > https://blog.jessfraz.com/post/the-art-of-closing/ ). Again, if this is > something that’s important to you, and you’re passionate about it, we’re > happy to provide further details about how you can directly contribute to > those efforts, so that at some point, in the future, it may be easier to > support what you want it to be. Thanks for this excellent information about the status of the Incident Reports policy section, it really was an eye-opener in what to expect from the CACC bugzilla list: it is used more as a central structured forum with closing capabilities, rather than as an ticketing system with deep and intermediate issue status tracking. Ryan, thank you for your in-depth response. -Matthias _______________________________________________ dev-security-policy mailing list [email protected] https://lists.mozilla.org/listinfo/dev-security-policy
