Thank you George, this is indeed interesting. As you and Ryan have noted, there is a lot of human interaction required, but I would like to find ways to better automate some of the work involved in managing incident bugs, especially since there seems to be no end of them in sight.
Meanwhile, I have reviewed and updated the bugs that you pointed out. - Wayne On Mon, Feb 18, 2019 at 2:35 PM Ryan Sleevi via dev-security-policy < dev-security-policy@lists.mozilla.org> wrote: > On Sat, Feb 16, 2019 at 12:24 PM George Macon via dev-security-policy < > dev-security-policy@lists.mozilla.org> wrote: > > > Prompted by the update to the Incident Response guidance earlier this > > week, I wondered how well CAs are doing at giving timely updates on the > > CA Incident bugs. > > > > I put together a quick prototype [0] that would examine the bug > > whiteboards and last-update timestamps and report those where the next > > update date is in the past. You can see the results [1] if you are > > curious. I examined a couple of the bugs identified to see what the > > status was: > > > Thanks! It’s very interesting to see your results! > > For what it’s worth, I don’t think this is a particularly valuable metric. > It’s certainly one we (peers) have discussed, but it’s not one that is a > way to measure the back and forth that goes on with CAs. > > In particular, the Next Update is only set when there is a concrete and > committed plan - and more issues than most do not catch that, because > there’s substantial discussion to get to that point, and CAs aren’t doing > particularly well there. It also doesn’t capture when the Mozilla peers > themselves are swamped by issues - for example, the 68 outstanding issues > by CAs. While distrusting CAs is certainly a path to ensuring less risk of > overload, it’s perhaps a bit more than CAs with outstanding incidents may > feel is desirable. > > As you note, it presently involves a rather large amount of human > interaction to process. One way that members in the community can help is > to engage on these issues by asking questions if there is more information > needed. I would ask that members refrain from sharing their own conclusions > on the bugs - and certainly believe it may be inappropriate to try and > answer questions if they are not the CA in question (as some participants > are inclined to do) - but there’s huge benefit in encouraging more and > meaningful details and highlighting similarities that may exist. > > > > > > > 1) https://bugzilla.mozilla.org/show_bug.cgi?id=1523680 > > > > The whiteboard does not contain a Next Update. The last action was the > > CA providing the incident report on 2019-02-01. If there are no > > questions or comments about the incident, I think this bug can be closed. > > > > 2) https://bugzilla.mozilla.org/show_bug.cgi?id=1518560 > > > > The whiteboard Next Update was set to 2019-02-05 on 2019-01-22. The CA > > provided an update on 2019-02-04. This is before the stated Next Update > > but was more than a week ago. If there are no further questions or > > comments about the incident, I think this bug can be closed. > > > > 3) https://bugzilla.mozilla.org/show_bug.cgi?id=1524143 > > > > The whiteboard does not contain a Next Update. The last action was an > > email to the CA requesting an incident report on 2019-01-31. The CA is > > not in compliance with the guidelines. > > > > 4) https://bugzilla.mozilla.org/show_bug.cgi?id=1495524 > > > > The whiteboard Next Update was set to 2019-01-01 on 2018-10-03. The CA > > has not commented in the bug since then. The CA is not in compliance > > with the guidelines. > > > > 5) https://bugzilla.mozilla.org/show_bug.cgi?id=1492006 > > > > The whiteboard Next Update was set to 2019-01-24 on 2019-01-17 following > > a comment by the CA that they would provide an update by 2019-01-24. The > > CA has not commented in the bug since then. The CA is not in compliance > > with the guidelines. > > > > I had hoped that a tool could automatically identify when CAs are not in > > compliance with the guidelines, but it looks like that determination > > actually requires reading and understanding the bug history. Oh, well. > > > > -George Macon > > > > [0]: https://gitlab.com/gmacon/moz-ca-incident-update > > [1]: https://gitlab.com/snippets/1822917 > > _______________________________________________ > > dev-security-policy mailing list > > dev-security-policy@lists.mozilla.org > > https://lists.mozilla.org/listinfo/dev-security-policy > > > _______________________________________________ > dev-security-policy mailing list > dev-security-policy@lists.mozilla.org > https://lists.mozilla.org/listinfo/dev-security-policy > _______________________________________________ dev-security-policy mailing list dev-security-policy@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-security-policy
