Incident Report - Entrust did not revoke all certificates with underscore characters before the 15 January 2019 dealine
On January 18, 2019, Entrust discovered that 9 SSL certificates with underscore characters which were issued for more than 30 days were not revoked before 15 January 2019. All certificates were revoked on 18 January 2019. Details of the incident report can be found here, https://bugzilla.mozilla.org/show_bug.cgi?id=1521520. Thanks, Bruce. ___ dev-security-policy mailing list dev-security-policy@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-security-policy
Re: Underscore characters
On Thu, Dec 27, 2018 at 8:22 PM Ryan Sleevi via dev-security-policy < dev-security-policy@lists.mozilla.org> wrote: > > I can't speak for Mozilla here, but I tried to lay out some clear > expectations: > 1) This is an extension of an existing incident, rather than treating it as > an exception to some long-standing or new rule > 2) This is being treated as part of the remediation (revocation) plan, > rather than as an intentional violation of some other requirement > This framing is technically correct and quite helpful in this situation, but I am concerned about what it appears to imply for other ambiguous requirements. Seeking clarity through a discussion and vote, as happened here, is more orderly than a sudden declaration that a practice is forbidden. This particular situation is a bit like a law that is challenged in court, then upheld by the court and enforced, as opposed to a new law being retroactively enforced. > 3) Going forward, "they weren't prepared for revocation" is not really an > acceptable answer in and of itself, and for this particular incident, > concrete proposals for how "They weren't prepared for revocation" can be > addressed or mitigated go a long way to addressing the underlying root > cause here, and by proxy, demonstrate a healthy awareness of and balancing > of risk, and ways to concretely mitigate that for the future. > > Yes, this is the desired outcome. Thanks James, Jeremy, Matt, and Ryan - I really appreciate the ideas that have come out in this discussion. I'll be looking at ways to use some of this thinking to improve the Mozilla incident reporting process - suggestions are welcome. - Wayne ___ dev-security-policy mailing list dev-security-policy@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-security-policy
Re: Underscore characters
On Fri, Dec 28, 2018 at 03:19:19AM +, Jeremy Rowley via dev-security-policy wrote: > > I'm not sure I'd call it "leniency", but I think you're definitely asking > > for "special treatment" -- pre-judgment on a potential incident so you can > > decide whether or not it's worth it (to DigiCert) to deliberately break the > > rules. > > I'm not sure there's a policy against asking for special treatment or > pre-judgment. Like I said, I feel like this is a weird area where I'm not > 100% > sure how to proceed. There's certainly a fuzzy area in the middle between "here is a problem, what should we do?" and the other extreme of "please let me know in advance if we'll be OK with doing this bad thing, because I'd like to decide whether it's worth breaking the rules". I have to say that several of your messages have read far more towards the latter than the former. Of course, the ability to distinguish is muddied by the need for you to provide specific data about the scope of the problem, which focuses things on just DigiCert, when there is the distinct possibility that other CAs are sitting quietly in the wings, having all the data but not wanting to step into the ring, as it were. > Like how do you raise when you think obedience to rules > is riskier than breaking them? Breaking them then explaining why seems like a > really bad idea. The best I could come up with is ask what to do and see if > the browsers agree. Acknowledged that this would be very bad in most cases, > but I'm not sure where you decide? I think you've followed the best course open to you. Talking about issues is pretty much guaranteed to be better than keeping quiet and hoping for the best (thanks, CT!). Certainly, knowingly breaking the rules and then having it turn up later is terrible -- as Ryan said, that's a quick way to get yourself distrusted. I certainly think that if any other CA comes out with an incident report post-Jan-15 dealing with unrevoked underscore-bearing certificates, the general reaction is going to be along the lines of, "are you *kidding* me?!?". > > What were the criteria by which DigiCert decided which customers to grant > > exceptions to? [snip] > Honestly, it came down to which ones were the most mad at me for telling > them I am going to revoke their certs. I can imagine... > > First off, your customers. There is a certain amount of exposition in the > > pharmacy company bug, however I can't say that what's there so far fills me > > with a sense of contentment. You said in your most recent post, "Security > > vulnerabilities are patched based on their rating", and that lacking a CVSS > > it is difficult to get recognition of a problem. Would it be fair to say > > that this narrow approach to security is shared by all/most/some/none of > > the > > other similarly situated customers? > > No, but it's generally how people can get exceptions to the blackout period. > More the norm is around how these certs are rolled out. They fall under three > camps: a) a third party offering the main companies service that requires a > bunch of testing and permissions (probably contractual), b) complicated > policies about changes during/around blackout periods and c) certs actually > used in software that require code changes and deployment to update. Those are useful categories to have, thanks. It's especially handy for CAs to bear in mind when they're communicating with their customers about the risks of deeply embedding data which may need to change at short notice. > > Focusing on the "what about next time?" aspect, which I believe is the most > > important, I'd be interested to know what your customers are planning on > > changing about their systems and processes, such that if a similar event > > happens in the future, the outcome won't be the same. > > After this, I'd like to talk about removing some of the Symantec roots from > Mozilla. A lot of these don't need trust in Mozilla and Chrome. The mix is in > the OS vs. Web ecosystem. They need trust in OS platforms, but Web is more > optional for a lot of the certs. If we have roots that are only trusted in > the two OS platforms (MS and Apple), the risk changes for the web community. I wonder how well that'll work out, given the dominant server platform (Linux, in its many and varied incarnations) generally sources its trust store from Mozilla (for better or worse). Given the highly variable timeline that distros have for updating their trust stores, you might be dealing with the fallout from that one for a *long* time to come. > > Hence, what is it that DigiCert plans to change, such that an equivalent > > result cannot happen in the future, given a similar event? There was one > > rather draconian possibility suggested up-thread, of DigiCert limiting > > itself to 100 days validity, and revoking a number of randomly-chosen > > certificates periodically. That would certainly remove any practical > > possibility
RE: Underscore characters
>> I think Matt provided a pretty clear moral hazard here - of customers >> suggesting their CAs didn't do enough (e.g. should have tried harder to >> intentionally violated by not revoking). One significant way to mitigating >> that risk is to take meaningful steps to ensure that "We couldn't revoke" is >> not really a viable or defensible option. Oh – thanks. I missed that. A lack of knowledge is already not a defensible position. Revocation requirements and an agreement to revoke within 24 hours is in all our of existing DigiCert contracts. The same language is going into all Symantec customer contracts now as customers transition to DigiCert systems. All of our documentation, including the CPS, say we can revoke with less than 1 day notice. >From section 4.9.1 of our CPS: DigiCert will revoke a Certificate within 24 hours if one or more of the following occurs: 1. The Subscriber requests in writing that DigiCert revoke the Certificate; 2. The Subscriber notifies DigiCert that the original Certificate request was not authorized and does not retroactively grant authorization; 3.DigiCert obtains evidence that the Subscriber’s Private Key corresponding to the Public Key in the Certificate suffered aKey Compromise; or 4. DigiCert obtains evidence that the validation of domain authorization or control for any FDQN or IP address in the Certificate should not be relied upon. DigiCert may revoke a certificate within 24 hours and will revoke a Certificate within 5 days if one or more of the following occurs: 1. The Certificate no longer complies with the requirements of Sections 6.1.5 and 6.1.6 of the CA/B forum baseline requirements; 2. DigiCert obtains evidence that the Certificate was misused; 3.The Subscriber or the cross‐certified CA breached a material obligation under the CP, this CPS, or the relevant agreement; 4. DigiCert confirms any circumstance indicating that use of a FQDN or IP address in the Certificate is no longer legally permitted (e.g. a court or arbitrator has revoked a Domain Name registrant’s right to use the Domain Name, a relevant licensing or services agreement between the Domain Name registrant and the Applicant has terminated, or the Domain Name registrant has failed to renew the Domain Name); 5. DigiCert confirms that a Wildcard Certificate has been used to authenticate a fraudulently misleading subordinate FQDN; 6. DigiCert confirms a material change in the information contained in the Certificate; 7. DigiCert confirms that the Certificate was not issued in accordance with the CA/B forum requirements or the DigiCert CP or this CPS; 8. DigiCert determines or confirms that any of the information appearing in the Certificate is inaccurate; 9. DigiCert’s right to issue Certificates under the CA/B forum requirements expires or is revoked or terminated, unless DigiCert has made arrangements to continue maintaining the CRL/OCSP Repository; …. This is why I couch it as we can revoke technically and legally, but I don’t think we should. >> This doesn't really inspire confidence. If the answer for how to deal with >> this is block efforts to remediate issues, then it runs all the risk that >> Matt was speaking to. "We knew people couldn't replace in January" is a >> problem, for sure, but because fundamentally the risk is always there that >> someone would need to revoke in January - or December, or November, or >> whenever the sensitive holiday freeze or critical sales or lunar alignment >> or personal vacation is - it's not really a mitigation at all for the issue. >> I tried to give suggestions earlier for meaningful steps - such as making >> sure all customers know that certificates may need to be revoked as soon as >> 24 hours. This has been a pattern of challenge in the past for DigiCert if I >> recall correctly - I believe both Blizzard and GitHub had issues where the >> keys were compromised, but these organizations didn't want to revoke the >> certs until they could ship new private keys in their software (... ignoring >> all the issues in that one). I know you've said you've got the contracts in >> place to defensibly revoke these, but how are you helping your users >> understand these risks? Do you have documentation on this? Do you recommend >> users use automation? I know some of this speaks to business practice, but I >> think that's somewhat core to the issue - since revocation may be required, >> how is the CA, the party best placed to communicate to the customer, >> communicating that necessity? Sorry – I thought you meant in addition to those things. All customers know we can revoke within 24 hours. Note that in both those cases the GitHub case we did revoke the cert within 24 hours of notification. We have documentation on revocation (eg https://www.digicert.com/certificate-revocation.htm) and talk about it a lot. We also recommend
Re: Underscore characters
On Thu, Dec 27, 2018 at 10:00 PM Jeremy Rowley wrote: > The risk Matt identified is too nebulous of an issue to address, tbh. How > do you address a moral issue? The only way I can think of to address the > moral issue is to say “we promise to be good”. But the weight that carries > depends on how much you trust the actor. If you trust the actor, then the > moral issue is addressed. If you don’t trust the actor, moral issue is not > addressed. If you or Matt can identify a specific threat you’d like me to > address about the moral issue, I’ll do my best to respond. > I think Matt provided a pretty clear moral hazard here - of customers suggesting their CAs didn't do enough (e.g. should have tried harder to intentionally violated by not revoking). One significant way to mitigating that risk is to take meaningful steps to ensure that "We couldn't revoke" is not really a viable or defensible option. > >- What happens is that you ask why there is risk of outage to begin >with and what can be done to improve going forward? Let’s assume you do >revoke, and it causes an outage - is DigiCert taking steps to ensure no >customer of theirs is ever faced with that risk? If so, what are those >steps? > > > > Yeah – there are several things we can do to improve going forward: > >1. Communicate better with the customers. The first mistake was >waiting until we had good data to communicate with the customers. This >delayed notification. This was unknown to me at the time, or we would have >sent out communication prior to the ballot passing. That instruction has >been passed along (no waiting on these critical issues) plus training. >2. No more skipping CAB Forum meetings for me. This was easily a >foreseeable issue because we knew people couldn’t replace in January. I >think it’s been brought up a half dozen times in the forum at least. I’m >not sure why we didn’t communicate this in Shanghai. But, the real problem >is I didn’t have direct knowledge of what was going on. I probably need to >be there in person each time so we can align the company correctly with >that is going on. > > That... doesn't really inspire confidence. If the answer for how to deal with this is block efforts to remediate issues, then it runs all the risk that Matt was speaking to. "We knew people couldn't replace in January" is a problem, for sure, but because fundamentally the risk is always there that someone would need to revoke in January - or December, or November, or whenever the sensitive holiday freeze or critical sales or lunar alignment or personal vacation is - it's not really a mitigation at all for the issue. I tried to give suggestions earlier for meaningful steps - such as making sure all customers know that certificates may need to be revoked as soon as 24 hours. This has been a pattern of challenge in the past for DigiCert if I recall correctly - I believe both Blizzard and GitHub had issues where the keys were compromised, but these organizations didn't want to revoke the certs until they could ship new private keys in their software (... ignoring all the issues in that one). I know you've said you've got the contracts in place to defensibly revoke these, but how are you helping your users understand these risks? Do you have documentation on this? Do you recommend users use automation? I know some of this speaks to business practice, but I think that's somewhat core to the issue - since revocation may be required, how is the CA, the party best placed to communicate to the customer, communicating that necessity? As Matt spoke to it somewhat, there's understandably competitive advantage to being the CA that will try their hardest not to revoke. And while I don't think this has risen to that level based on the information provided so far, understanding how that perception is being mitigated is key. There are other solutions, to be sure. Helping users move from publicly trusted CAs to managed CAs, for example, can still meet the business needs of these users w/o the attendant revocation risk. Things like Heartbleed have shown that rapid revocation can be necessary. Misissuance or misvalidation by the CA that results in revocation surely can as well. Understandably, an answer of "Don't ever misissue" is great, but if it's really pinning all the hopes on one thing. Other CAs have taken steps like ensuring automation and short-lived certs as a way of ensuring that the upper-bound of any issue is limited (for example, to 90 days, or six months), and that automation is the default way of getting certs. > >- And this is the framing that I think is incredibly helpful. >Understanding why customers can’t change, and what steps are being done to >ensure they can, is hugely useful. Wayne’s question were to this point - as >were mine towards understanding the problem from the other side, which are >steps the CA is taking. As I've repeatedly
RE: Underscore characters
> I don't think there's *any* result from all this that everyone would > consider desirable -- otherwise we wouldn't need to have this conversation. + 1 to that. > I'm not sure I'd call it "leniency", but I think you're definitely asking > for "special treatment" -- pre-judgment on a potential incident so you can > decide whether or not it's worth it (to DigiCert) to deliberately break the > rules. I'm not sure there's a policy against asking for special treatment or pre-judgment. Like I said, I feel like this is a weird area where I'm not 100% sure how to proceed. Like how do you raise when you think obedience to rules is riskier than breaking them? Breaking them then explaining why seems like a really bad idea. The best I could come up with is ask what to do and see if the browsers agree. Acknowledged that this would be very bad in most cases, but I'm not sure where you decide? > What were the criteria by which DigiCert decided which customers to grant > exceptions to? My default assumption is "whichever ones will cost us the > most money, on a risk-of-departure-weighted basis, if we revoke their > misissued certs", so if DigiCert's criteria was different, I'd be keen to > have my assumption changed. Based on the number of certificates, the reasons the customer identified they couldn't make change, and whether revocation would take down a critical site. It actually isn't tied to $ at all. The largest issuer of certificates isn't on the exception list. Honestly, it came down to which ones were the most mad at me for telling them I am going to revoke their certs. I also filed the incident reports in that order. > First off, your customers. There is a certain amount of exposition in the > pharmacy company bug, however I can't say that what's there so far fills me > with a sense of contentment. You said in your most recent post, "Security > vulnerabilities are patched based on their rating", and that lacking a CVSS > it is difficult to get recognition of a problem. Would it be fair to say > that this narrow approach to security is shared by all/most/some/none of the > other similarly situated customers? No, but it's generally how people can get exceptions to the blackout period. More the norm is around how these certs are rolled out. They fall under three camps: a) a third party offering the main companies service that requires a bunch of testing and permissions (probably contractual), b) complicated policies about changes during/around blackout periods and c) certs actually used in software that require code changes and deployment to update. > As an aside, on the subject of "there's no CVSS score for this", let me fix > that up, with the official WombleSecure(TM)(R)(Patent Pending) CVSS for > "your certs are getting revoked": https://clicktime.symantec.com/a/1/yUHHbekYeF5I1ApCiRHB3c4GRi5h119CZduhXSUjcHQ=?d=jjXJ8wGMEM-BgSpW3_vhyQL0sXCIhGbj3gBpMQofOamgauLb68trqD6rFgW1WlGMp2x8t2VFcaY0DBIxDVgeeB1NTgFMApldbJMcAgO-QzAYKleHGSG1QMDssL8YiuasGm7sy54zIql5pGoFC32z-FPTIi19g1UDgwcBY97oowWvIdYn96-dpAc9Bgo0beU6KZJB4GgT4nsTYZfQEPWR6iJovigq7cka80r2jfU6Ef-FnpegGAkDENlMwnIoHo4ti6V0kNC1BnXX92EeVaD_XCRNLlzHjHvbe0_9OrBDSAOuXH7r90tkFNs5Jf15Y9tnE-nNgpNo-7ATwrZ6C-AfpSHr9tX-RnCPFHoSUEIJ9az2IiiMo_si4rA2uaMaKtjN1Ziuk7XNO9s%3D=https%3A%2F%2Fwww.first.org%2Fcvss%2Fcalculator%2F3.0%23CVSS%3A3.0%2FAV%3AN%2FAC%3AL%2FPR%3AN%2FUI%3AN%2FS%3AU%2FC%3AN%2FI%3AN%2FA%3AH%2FE%3AH%2FRL%3AO%2FRC%3AC%2FAR%3AH%2FMAV%3AN%2FMAC%3AL%2FMPR%3AN%2FMUI%3AN%2FMS%3AU%2FMC%3AN%2FMI%3AN%2FMA%3AH 7.5 base, 7.2 temporal, and 8.9 environmental. All those scores are in the "high" band. "Availability" *is* one of the sides of the security triangle, after all. Lol - thanks. I'll be sure to share this with them. > Focusing on the "what about next time?" aspect, which I believe is the most > important, I'd be interested to know what your customers are planning on > changing about their systems and processes, such that if a similar event > happens in the future, the outcome won't be the same. After this, I'd like to talk about removing some of the Symantec roots from Mozilla. A lot of these don't need trust in Mozilla and Chrome. The mix is in the OS vs. Web ecosystem. They need trust in OS platforms, but Web is more optional for a lot of the certs. If we have roots that are only trusted in the two OS platforms (MS and Apple), the risk changes for the web community. > A similar question applies, even more forcefully, to DigiCert itself. > Clearly, whatever you've done so far didn't work, because these customers of > yours didn't heed whatever warnings and caveats you provided, and built > themselves systems and processes that are unable to comply with their > agreements to DigiCert (and, by extension, relying parties). See above. Also see my response to Ryan on the migration from legacy Symantec systems. > Hence, what is it that DigiCert plans to change, such that an equivalent > result cannot happen in the
RE: Underscore characters
is not punishment - but understanding how these issues are being addressed. The main blocker for all of these is policy, not technology. I don’t know how to solve third party policy decisions, which is why I can’t seem to answer the questions. The process of planning a change, getting sign-off, rolling the change to stage, getting more sign-off, and then rolling to production with final testing combined with the blackout periods is making something that should be easy very difficult. I run an agile team at DigiCert so none of these are concerns when we roll a change internally. It’s the revocation part that is getting people up in arms. The consistent message I’ve gotten from customers is that changing domains and certificates requires the same process. It’s just as fast to roll out a change to both items as change just a certificate. The built-in CAB Forum 30 day cert requirement isn’t solving the issue because of the way they roll changes, not because the 30 day certs aren’t available. * This seems like a significant improvement from “100% of customers can’t” Definitely an improvement. I’m hoping to get to 100% by the time we hit Jan 15th. The four I posted (and one more I got more info from today) probably won’t. Even within those customers, we’re asking them identify specifically which certificates cannot be replaced in time. * I mean, it’s two-fold, right? Any incident can lead to total distrust, but it’s also unlikely that a single incident leads to total distrust. The way to balance those competing statements is to do what you’re doing - and to be transparent. As Matt has highlighted, there’s a huge risk here that this leads to a moral hazard - and the best way to mitigate that is to discuss steps being taken to reduce that risk going forward, particularly about what a core part of the problem statement is - difficulty in revocation. This isn’t our first incident sadly ☹. It probably won’t be our last. The transition from Symantec to DigiCert was….rough. * In a number of ways, an unintentional violation is worse than an intentional violation. Ignorance is not really an excuse when you hold keys to the Internet, and being asleep at the wheel is hugely dangerous. So, if I had to pick between an intentional violation and an unintentional (and preventable) violation, I'd likely pick intentional. But there's also a huge hazard with intentional violations - those reveal potentially systemic issues and a lack of good faith, especially if they become common-place. We definitely saw CAs perform intentional violations and notify after-the-fact, and that's far, far worse than those that notify before intentionally violating (I think every post-facto notification for intentional incident has, eventually, lead to that CAs distrust). Totally agree. I really don’t want to violate the BRs, and this shouldn’t be the norm. I also recognize we don’t want to invite this question for every BR change. Maybe better Mozilla guidelines about what’s acceptable requests and what’s not? * So somewhere on the scale of things, we're in a better place than most every alternative. But to ensure this is in that 'good faith' side of things, understanding what the factors are that have been evaluated, and what steps are being taken to prevent this, are significant. As I said, I think the principles captured in https://wiki.mozilla.org/CA/Responding_To_An_Incident#Revocation and in the discussion about how at least some of us see this (that it's related to underscores incident response) suggests that it's not, in fact, the end of the world, or the CA, provided that meaningful data behind the decision to not revoke is given, meaningful plans and timelines for resolution are given, and meaningful steps to prevent this from ever happening again are given. It becomes an incident report, and the result is not a stern lecture - but concrete and quantifiable steps as to how to improve. Thanks Ryan. This post was really nice. Appreciate it. From: Ryan Sleevi Sent: Thursday, December 27, 2018 7:15 PM To: Jeremy Rowley Cc: James Burton ; Ryan Sleevi ; mozilla-dev-security-policy Subject: Re: Underscore characters On Thu, Dec 27, 2018 at 6:56 PM Jeremy Rowley mailto:jeremy.row...@digicert.com> > wrote: The risk is primarily outages of major sites across the web, including certs used in Google wallet. We’re thinking that is a less than desirable result, but we weren’t sure how the Mozilla community would feel/react. I don’t think that is a particularly helpful framing, to be honest. The risk these organizations face here is self-inflicted; regardless of the feeling of underscores, there is unquestionably an issue for organizations that cannot respond in the BR timeframes, let alone extended ones that extend for months. That's a real ecosystem issue, and regardless of the CA these customers partner with, an issue that needs b
Re: Underscore characters
On Thu, Dec 27, 2018 at 11:56:41PM +, Jeremy Rowley via dev-security-policy wrote: > The risk is primarily outages of major sites across the web, including > certs used in Google wallet. We’re thinking that is a less than desirable > result, but we weren’t sure how the Mozilla community would feel/react. I don't think there's *any* result from all this that everyone would consider desirable -- otherwise we wouldn't need to have this conversation. > We’re still considering revoking all of the certs on Jan 15th based on > these discussions. I don’t think we’re asking for leniency (maybe we are > if that’s a factor?) I'm not sure I'd call it "leniency", but I think you're definitely asking for "special treatment" -- pre-judgment on a potential incident so you can decide whether or not it's worth it (to DigiCert) to deliberately break the rules. > Normally, we would just revoke the certs, but there are a significant > number of certs in the Alexa top 100. We’ve told most customers, “No > exception”. What were the criteria by which DigiCert decided which customers to grant exceptions to? My default assumption is "whichever ones will cost us the most money, on a risk-of-departure-weighted basis, if we revoke their misissued certs", so if DigiCert's criteria was different, I'd be keen to have my assumption changed. > I also thought it’s better to get the information out there so we can all > make rational decisions (DigiCert included) if as many facts are known as > possible. There are a number of areas that I think could stand to have some more facts added. First off, your customers. There is a certain amount of exposition in the pharmacy company bug, however I can't say that what's there so far fills me with a sense of contentment. You said in your most recent post, "Security vulnerabilities are patched based on their rating", and that lacking a CVSS it is difficult to get recognition of a problem. Would it be fair to say that this narrow approach to security is shared by all/most/some/none of the other similarly situated customers? As an aside, on the subject of "there's no CVSS score for this", let me fix that up, with the official WombleSecure(TM)(R)(Patent Pending) CVSS for "your certs are getting revoked": https://www.first.org/cvss/calculator/3.0#CVSS:3.0/AV:N/AC:L/PR:N/UI:N/S:U/C:N/I:N/A:H/E:H/RL:O/RC:C/AR:H/MAV:N/MAC:L/MPR:N/MUI:N/MS:U/MC:N/MI:N/MA:H 7.5 base, 7.2 temporal, and 8.9 environmental. All those scores are in the "high" band. "Availability" *is* one of the sides of the security triangle, after all. Focusing on the "what about next time?" aspect, which I believe is the most important, I'd be interested to know what your customers are planning on changing about their systems and processes, such that if a similar event happens in the future, the outcome won't be the same. A similar question applies, even more forcefully, to DigiCert itself. Clearly, whatever you've done so far didn't work, because these customers of yours didn't heed whatever warnings and caveats you provided, and built themselves systems and processes that are unable to comply with their agreements to DigiCert (and, by extension, relying parties). Hence, what is it that DigiCert plans to change, such that an equivalent result cannot happen in the future, given a similar event? There was one rather draconian possibility suggested up-thread, of DigiCert limiting itself to 100 days validity, and revoking a number of randomly-chosen certificates periodically. That would certainly remove any practical possibility of customers not being able to refresh their certificates if-and-when, however I can imagine it might be a bit of a shock to the system for many of them. Hence, I'd be interested in hearing what DigiCert's actual plans are, because if it were my call, *that* would be the single biggest factor in determining the disposition of an event like this. That errors occur is regrettable, but it's when they happen repeatedly that it becomes indefensible. - Matt ___ dev-security-policy mailing list dev-security-policy@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-security-policy
Re: Underscore characters
On Thu, Dec 27, 2018 at 6:56 PM Jeremy Rowley wrote: > The risk is primarily outages of major sites across the web, including > certs used in Google wallet. We’re thinking that is a less than desirable > result, but we weren’t sure how the Mozilla community would feel/react. > I don’t think that is a particularly helpful framing, to be honest. The risk these organizations face here is self-inflicted; regardless of the feeling of underscores, there is unquestionably an issue for organizations that cannot respond in the BR timeframes, let alone extended ones that extend for months. That's a real ecosystem issue, and regardless of the CA these customers partner with, an issue that needs both better understanding and, to be honest, better prevention. Matt has spoken at length to the risk to the community, which doesn’t really seem like it’s been acknowledged, let alone proposed as to how it will be mitigated. I have to ask again - what steps is DigiCert taking to avoid these issues going forward? We’re still considering revoking all of the certs on Jan 15th based on > these discussions. I don’t think we’re asking for leniency (maybe we are > if that’s a factor?), but I don’t know what happens if you’re faced with > causing outages vs. compliance. > What happens is that you ask why there is risk of outage to begin with and what can be done to improve going forward? Let’s assume you do revoke, and it causes an outage - is DigiCert taking steps to ensure no customer of theirs is ever faced with that risk? If so, what are those steps? I started the conversation because I feel like we should be good netizans > and make people aware of what’s going on instead of just following policy. > I’m actually surprised at least one other CA that has issued a large number > of underscore character certs hasn’t run into the same timing issues. > This seems to suggest that perhaps other CAs have prepared their customers for revocation. How does this surprise - that no other CA faces this - lead to tangible changes in the business processes? How would this change, if another CA did have the same issue? Surely you can see there are real and fundamental issues that you’re uniquely qualified to help your customers address in ways that we cannot. Have you analyzed CT, for example, to see why DigiCert is unique? Certainly, by sheer volume, it's heavily tilted towards the old Symantec infrastructure - and the customers that came over to DigiCert. With those sorts of details, how does this change how things were done, or how they will be done? I’m not trying to pick on y’all - I think it is legitimately good that you provided concrete data. Even if you do revoke on Jan 15, this is still useful to understand the challenges, but only if this leads to meaningful changes. What might those look like? Normally, we would just revoke the certs, but there are a significant > number of certs in the Alexa top 100. We’ve told most customers, “No > exception”. I also thought it’s better to get the information out there so > we can all make rational decisions (DigiCert included) if as many facts are > known as possible. > And this is the framing that I think is incredibly helpful. Understanding why customers can’t change, and what steps are being done to ensure they can, is hugely useful. Wayne’s question were to this point - as were mine towards understanding the problem from the other side, which are steps the CA is taking. As I've repeatedly highlighted from https://wiki.mozilla.org/CA/Responding_To_An_Incident#Revocation , the goal is not punishment - but understanding how these issues are being addressed. > > We are working with the partners to get the certs revoked before the > deadline. Most will. > This seems like a significant improvement from “100% of customers can’t” By January 15th, I hope there won’t be too many certs left. Unfortunately, > by then it’s also too late to discuss what happens if the cert is not > revoked. Ie – what are the benefits of revoking (strict compliance) vs > revoking the larger impact certs as they are migrated (incident report). > Unfortunately part 2, there’s no guidance on whether an incident report > means total distrust v. something on your audit and a stern lecture. > I mean, it’s two-fold, right? Any incident can lead to total distrust, but it’s also unlikely that a single incident leads to total distrust. The way to balance those competing statements is to do what you’re doing - and to be transparent. As Matt has highlighted, there’s a huge risk here that this leads to a moral hazard - and the best way to mitigate that is to discuss steps being taken to reduce that risk going forward, particularly about what a core part of the problem statement is - difficulty in revocation. I’d happily suffer a lecture than take down a top site. Not so willing to > gamble the whole company. This is why we wanted to have the discussion now, > despite no violation so far. The response from the browsers
RE: Underscore characters
Treading carefully… Mozilla is the only browser related to the discussion. Probably sufficient to say that the revocation/no-revoke decision is entirely dependent on the results of this thread. From: James Burton Sent: Thursday, December 27, 2018 6:07 PM To: Jeremy Rowley Cc: Matt Palmer ; mozilla-dev-security-policy Subject: Re: Underscore characters I'm not sure if you're allowed to state this publicly. Has Microsoft giving you the go ahead? On Fri, Dec 28, 2018 at 1:05 AM Jeremy Rowley via dev-security-policy mailto:dev-security-policy@lists.mozilla.org> > wrote: I disagree that we won't get that. I think we could see a "it's okay to wait until April 30 for large pharmacy" or "Waiting until April 30 is too long but March 1 is okay". I don't think Mozilla wants outages either. But... if Mozilla did say that we should revoke now, that would be great as well. I'd have a firm answer I can go back with. No risk, but no exception. Well except moral risk of course -Original Message- From: dev-security-policy mailto:dev-security-policy-boun...@lists.mozilla.org> > On Behalf Of Matt Palmer via dev-security-policy Sent: Thursday, December 27, 2018 5:55 PM To: dev-security-policy@lists.mozilla.org <mailto:dev-security-policy@lists.mozilla.org> Subject: Re: Underscore characters On Fri, Dec 28, 2018 at 12:12:03AM +, Jeremy Rowley via dev-security-policy wrote: > This is very helpful. If I had those two options, we'd just revoke all > the certs, screw outages. Unfortunately, the options are much broader than that. > If I could know what the risk v. benefit is, then you can make a > better decision? DigiCert distrusted - all revoked. DigiCert gets some > mar on its audit - outages seem worse. Make sense? Given that Mozilla wants CAs to abide by its policies, which include adherence to the BRs, and you appear to be saying that you'll adhere to the BRs if you're threatened with distrust... I'd say the logical response from Mozilla would be to threaten distrust. I doubt, especially now, that you'll get a categorical advance "it's OK to not revoke" from Mozilla. - Matt ___ dev-security-policy mailing list dev-security-policy@lists.mozilla.org <mailto:dev-security-policy@lists.mozilla.org> https://clicktime.symantec.com/a/1/JAUY6LMmpzDeGtxtOiXLJVWWYjWV65xcMjKoLj_GS <https://clicktime.symantec.com/a/1/JAUY6LMmpzDeGtxtOiXLJVWWYjWV65xcMjKoLj_GSgs=?d=2r4BCPONnLRAQaYxhIYsrR2xI_C73HdzeRvSzxfwF1rOccA0cfq95qcKptTpNVYkGzCfglu40QMyhwHQJyWghm9tDreLIrUFB4D0ugqZlnn2SKyEI85b9QcQlb6I-o78NypjSLQRAUF9s9i5tFsXc6oVsnhZly7GCR8HrTZqfLEL8fXQKwA8A7MRCYPr2Hy61TCorYztrVr2u8IME1WcJdVQxd1tkB> gs=?d=2r4BCPONnLRAQaYxhIYsrR2xI_C73HdzeRvSzxfwF1rOccA0cfq95qcKptTpNVYkGzCfgl u40QMyhwHQJyWghm9tDreLIrUFB4D0ugqZlnn2SKyEI85b9QcQlb6I-o78NypjSLQRAUF9s9i5tF sXc6oVsnhZly7GCR8HrTZqfLEL8fXQKwA8A7MRCYPr2Hy61TCorYztrVr2u8IME1WcJdVQxd1tkB MIgZG8M74du5AO2ELfvkGfV3pBYbOUubjwoFhmqqgsHy5GyDIO_EZS68OavUwfNHvpkZ-5paTSWR yGwQFw0uz8CKa2kO0IOOBGt55A-WAyvJnhPJScUvwu_c9n2KmEljO7EbvvYGYA0E3Ef6rWWdpZbm D8FZ39LChfaUgdEP4DX6Y%3D=https%3A%2F%2Flists.mozilla.org <http://2Flists.mozilla.org> %2Flistinfo%2Fdev- security-policy ___ dev-security-policy mailing list dev-security-policy@lists.mozilla.org <mailto:dev-security-policy@lists.mozilla.org> https://lists.mozilla.org/listinfo/dev-security-policy smime.p7s Description: S/MIME cryptographic signature ___ dev-security-policy mailing list dev-security-policy@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-security-policy
RE: Underscore characters
I disagree that we won't get that. I think we could see a "it's okay to wait until April 30 for large pharmacy" or "Waiting until April 30 is too long but March 1 is okay". I don't think Mozilla wants outages either. But... if Mozilla did say that we should revoke now, that would be great as well. I'd have a firm answer I can go back with. No risk, but no exception. Well except moral risk of course -Original Message- From: dev-security-policy On Behalf Of Matt Palmer via dev-security-policy Sent: Thursday, December 27, 2018 5:55 PM To: dev-security-policy@lists.mozilla.org Subject: Re: Underscore characters On Fri, Dec 28, 2018 at 12:12:03AM +, Jeremy Rowley via dev-security-policy wrote: > This is very helpful. If I had those two options, we'd just revoke all > the certs, screw outages. Unfortunately, the options are much broader than that. > If I could know what the risk v. benefit is, then you can make a > better decision? DigiCert distrusted - all revoked. DigiCert gets some > mar on its audit - outages seem worse. Make sense? Given that Mozilla wants CAs to abide by its policies, which include adherence to the BRs, and you appear to be saying that you'll adhere to the BRs if you're threatened with distrust... I'd say the logical response from Mozilla would be to threaten distrust. I doubt, especially now, that you'll get a categorical advance "it's OK to not revoke" from Mozilla. - Matt ___ dev-security-policy mailing list dev-security-policy@lists.mozilla.org https://clicktime.symantec.com/a/1/JAUY6LMmpzDeGtxtOiXLJVWWYjWV65xcMjKoLj_GS gs=?d=2r4BCPONnLRAQaYxhIYsrR2xI_C73HdzeRvSzxfwF1rOccA0cfq95qcKptTpNVYkGzCfgl u40QMyhwHQJyWghm9tDreLIrUFB4D0ugqZlnn2SKyEI85b9QcQlb6I-o78NypjSLQRAUF9s9i5tF sXc6oVsnhZly7GCR8HrTZqfLEL8fXQKwA8A7MRCYPr2Hy61TCorYztrVr2u8IME1WcJdVQxd1tkB MIgZG8M74du5AO2ELfvkGfV3pBYbOUubjwoFhmqqgsHy5GyDIO_EZS68OavUwfNHvpkZ-5paTSWR yGwQFw0uz8CKa2kO0IOOBGt55A-WAyvJnhPJScUvwu_c9n2KmEljO7EbvvYGYA0E3Ef6rWWdpZbm D8FZ39LChfaUgdEP4DX6Y%3D=https%3A%2F%2Flists.mozilla.org%2Flistinfo%2Fdev- security-policy smime.p7s Description: S/MIME cryptographic signature ___ dev-security-policy mailing list dev-security-policy@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-security-policy
RE: Underscore characters
The 7 required items under the Mozilla template are: 1. Timeline of events 2. Timeline of actions taken 3. Whether the CA has stopping issuing 4. Summary of problematic certs 5. Cert data 6. How mistakes were made 7. Remediation plan The info we’re working on getting a complete list of: 1. Blackout periods 2. Where each cert is used in the infrastructure 3. Why 30 day certs won’t work (on a per cert basis) 4. Reason the certs are publicly trusted 5. What risk are associated with the replacement 6. The date each cert can be revoked Mostly we’re hearing back general answers. They’re almost all the same answer, but I’m really trying to get the level of detail requested. I see how you could interpret the question that way. I see it more as the CAB forum got the date wrong. Could Mozilla please extend this after weighing the risks of revoking vs. non-revoking? Maybe two sides of the same question. The second deadline is coming from the impacted parties. That’s the request from them so I’m relaying it on. Everyone is willing to move, just a matter of timing. If there’s a better balance of risk vs. risk, then we’d be happy to hear that. >> So the assumption here is that, in all of this discussion, DigiCert's done >> everything it can to understand the issue, the >> timelines, remediation, etc, and has plans to address both each and every >> customer and the systemic issues that have >> emerged. If that's not the case, then how are we not in one of those two >> scenarios above? And if it is the case, isn't that >> information readily available by now? The information is readily available for the companies I posted in incident reports, particularly the first one. I think we’ve done everything reasonable to understand the issue. I haven’t, for example, chartered a flight to sit in their data center and examine their infrastructure. We do have daily calls with most of them on the issue. Maybe the amount of information the company has provided should be the guiding light? From: Ryan Sleevi Sent: Thursday, December 27, 2018 1:16 PM To: Jeremy Rowley Cc: mozilla-dev-security-policy Subject: Re: Underscore characters I'm not trying to throw you under the bus here, but I think it's helpful if you could highlight what new information you see being required, versus that which is already required. I think, yes, you're right that it's not well received if you go violate the BRs and then, after the fact, say "Hey, yeah, we violated, but here's why", and finding out that the reasons are met with a lot of skepticism and the math being shaky, and you can see that from past incident reports it doesn't go over well. But it's also not well received if it's before, and the statement is "Our customer thinks we should violate the BRs. What would happen if we did, and what information do you need from us?". That gets into the moral hazard that Matt spoke to, and is a huge burden on the community where the expectation is that the CA says "Sorry, we can't do that". So the assumption here is that, in all of this discussion, DigiCert's done everything it can to understand the issue, the timelines, remediation, etc, and has plans to address both each and every customer and the systemic issues that have emerged. If that's not the case, then how are we not in one of those two scenarios above? And if it is the case, isn't that information readily available by now? >From the discussions on the incident reports, I feel like that's been the >heart of the questions; which is trying to understand what the root cause is >and what the remediation plan is. The statement "We'll miss the first >deadline, but we'll hit the second", but without any details about how or why, >or the steps being taken to ensure no deadlines are missed in the future, >doesn't really inspire confidence, and is exactly the same kind of feedback >that would be given post-incident. On Thu, Dec 27, 2018 at 1:50 PM Jeremy Rowley via dev-security-policy mailto:dev-security-policy@lists.mozilla.org> > wrote: There's a little bit of a "damned if you do, damned if you don't problem here". Wait until you have all the information? That's a paddlin'. File before you have enough information? That's a paddlin'. I'd appreciate better guidance on what Mozilla expects from these incident reports timing-wise. -Original Message- From: dev-security-policy mailto:dev-security-policy-boun...@lists.mozilla.org> > On Behalf Of Jeremy Rowley via dev-security-policy Sent: Thursday, December 27, 2018 11:47 AM To: r...@sleevi.com <mailto:r...@sleevi.com> Cc: dev-security-policy@lists.mozilla.org <mailto:dev-security-policy@lists.mozilla.org> Subject: RE: Underscore characters The o
Re: Underscore characters
On Fri, Dec 28, 2018 at 12:12:03AM +, Jeremy Rowley via dev-security-policy wrote: > This is very helpful. If I had those two options, we'd just revoke all the > certs, screw outages. Unfortunately, the options are much broader than that. > If I could know what the risk v. benefit is, then you can make a better > decision? DigiCert distrusted - all revoked. DigiCert gets some mar on its > audit - outages seem worse. Make sense? Given that Mozilla wants CAs to abide by its policies, which include adherence to the BRs, and you appear to be saying that you'll adhere to the BRs if you're threatened with distrust... I'd say the logical response from Mozilla would be to threaten distrust. I doubt, especially now, that you'll get a categorical advance "it's OK to not revoke" from Mozilla. - Matt ___ dev-security-policy mailing list dev-security-policy@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-security-policy
RE: Underscore characters
This is very helpful. If I had those two options, we'd just revoke all the certs, screw outages. Unfortunately, the options are much broader than that. If I could know what the risk v. benefit is, then you can make a better decision? DigiCert distrusted - all revoked. DigiCert gets some mar on its audit - outages seem worse. Make sense? -Original Message- From: dev-security-policy On Behalf Of thomas.gh.horn--- via dev-security-policy Sent: Thursday, December 27, 2018 1:50 PM To: mozilla-dev-security-pol...@lists.mozilla.org Subject: Re: Underscore characters As to why these certificates have to be revoked, you should see this the other way round: as a very generous service of the community to you and your customers! Certificates with (pseudo-)hostnames in them are clearly invalid, so a conforming implementation should not accept them for anything and they should not pose any security risk. Based on this assessment (no revokation if no security risk), a CA could very well issue a certificate including any of the (psuedo-)hostnames "example.com_cvs.com", "example.com/cvs.com", "cvs.com/example.com", "https://clicktime.symantec.com/a/1/Bz3KjBhWfzAsIJ0uIM5iJZb_Vq9KOZqIbbEqrWx1 PPc=?d=nuBPRsMXvpmDCViEfj_vdMTuPr8sqLAI5iKEWF4ohV9p1yKSHaat1UnUMwQC2TM1Glbqm sZ5vll_Ws-lffmZiGXLoAjAa1j4xYlIvj_mjSSwyyAqosT8up883sRCNtFds_0zcjRxOOoj2-Clo cugotsEOb5kZj4DN2uJO-MXnpA-ayZPZSvrBhJ61IzJdnfMh1ufcgt0H6eS4MDVVELwAzREz5sDF lQhRCO_bmD3I3jI7vj9qUbLzQFJGYVKa0aQ_RlnmWxfRFD0s4bJcUeW2SLinms3T2PnVDt62TguH hnVQeT7XLb0uAGF0x7KNhbpJbykznPGT6vDGP6xnntYiQHZgZqRiOfJvYE642rqp3X9NoRx26Q0Q Qy4KgOGUE-nAs60vFYry1msFrinKGViW9Q%3D=https%3A%2F%2Fexample.com%2Fcvs.com" , "example@cvs.com" to the owner of example.com (who, arguably, has the exact same right to them as the owner of cvs.com has) and refuse to revoke them. As to the consequences (in case this really becomes an incident report/incident reports): this shows a SEVERE lack of ability to revoke certificates on DigiCert's side, which must have been known AND ACCEPTED for a long time (this cannot be the first "blackout period" of (in the best case) 3.5 months). Thus, it seems to be a good idea to: 1. Henceforth, make NSS only accept certificates by DigiCert with a maximum validity of 100 days. Let's Encrypt has shown that this is clearly feasible. or 2. Henceforth, require DigiCert to revoke a small, randomly (e.g., using RFC 3797) selected subset of their certificates every day (within 7 days). If this, e.g., for the same reasons as outlined in these incident reports, is not possible, it will trigger (a incrementally decreasing number of) more incident reports. Both proposals would lead to more automation and a better understanding of the requirement of timely revocation, while pushing the ecosystem in the right direction. For its easiness, the first proposal would be my favorite but I would be very interested in hearing other people's thoughts about these proposals. ___ dev-security-policy mailing list dev-security-policy@lists.mozilla.org https://clicktime.symantec.com/a/1/2hiT00ldRBQieEaN_06CurvCo04Hq3RsaRxAAoyWN IY=?d=nuBPRsMXvpmDCViEfj_vdMTuPr8sqLAI5iKEWF4ohV9p1yKSHaat1UnUMwQC2TM1Glbqms Z5vll_Ws-lffmZiGXLoAjAa1j4xYlIvj_mjSSwyyAqosT8up883sRCNtFds_0zcjRxOOoj2-Cloc ugotsEOb5kZj4DN2uJO-MXnpA-ayZPZSvrBhJ61IzJdnfMh1ufcgt0H6eS4MDVVELwAzREz5sDFl QhRCO_bmD3I3jI7vj9qUbLzQFJGYVKa0aQ_RlnmWxfRFD0s4bJcUeW2SLinms3T2PnVDt62TguHh nVQeT7XLb0uAGF0x7KNhbpJbykznPGT6vDGP6xnntYiQHZgZqRiOfJvYE642rqp3X9NoRx26Q0QQ y4KgOGUE-nAs60vFYry1msFrinKGViW9Q%3D=https%3A%2F%2Flists.mozilla.org%2Flis tinfo%2Fdev-security-policy smime.p7s Description: S/MIME cryptographic signature ___ dev-security-policy mailing list dev-security-policy@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-security-policy
RE: Underscore characters
This is accurate. We have the technical capability and policy ability to revoke the certificates. What we were hoping was a discussion based on impact of the revocation so we could hear what we should do. Blind obedience isn't my favorite answer, but it's an option. The guidance so far is file an incident report now so we can discuss the potential impact. I've filed for two companies, crossed a couple more off the list, and am still working with the remainder to get things resolved. Although some have escalated over my head, I think most are eager to hear what the community has to say. I also think this is an interesting question for Mozilla's policy - not sure we've ever addressed a potential non-compliance like this. -Original Message- From: dev-security-policy On Behalf Of Peter Bowen via dev-security-policy Sent: Thursday, December 27, 2018 2:19 PM To: thomas.gh.h...@gmail.com Cc: mozilla-dev-security-pol...@lists.mozilla.org Subject: Re: Underscore characters On Thu, Dec 27, 2018 at 12:53 PM thomas.gh.horn--- via dev-security-policy < dev-security-policy@lists.mozilla.org> wrote: > > As to why these certificates have to be revoked, you should see this > the other way round: as a very generous service of the community to > you and your customers! > > Certificates with (pseudo-)hostnames in them are clearly invalid, so a > conforming implementation should not accept them for anything and they > should not pose any security risk. Based on this assessment (no > revokation if no security risk), a CA could very well issue a > certificate including any of the (psuedo-)hostnames > "example.com_cvs.com", "example.com/cvs.com", "cvs.com/example.com", "https://example.com/cvs.com;, "example@cvs.com" > to the owner of example.com (who, arguably, has the exact same right > to them as the owner of cvs.com has) and refuse to revoke them. > I'm not clear how you get that the owner of example.com is covered anywhere here. Parsed into labels, these all have com as the label closet to the root and then have 'com_cvs', 'com/cvs', 'com/example', 'com/cvs', and 'com@cvs' as the next label respectively. None have 'example' as the next label. > As to the consequences (in case this really becomes an incident > report/incident reports): this shows a SEVERE lack of ability to > revoke certificates on DigiCert's side, which must have been known AND > ACCEPTED for a long time (this cannot be the first "blackout period" > of (in the best > case) 3.5 months). I don't see how this follows. DigiCert has made it clear they are able to technically revoke these certificates and presumably are contractually able to revoke them as well. What is being said is that their customers are asking them to delay revoking them because the _customers_ have blackout periods where the customers do not want to make changes to their systems. DigiCert's customers are saying that they are judging the risk from revocation is greater than the risk from leaving them unrevoked and asking DigiCert to not revoke. DigiCert is then presenting this request along to Mozilla to get feedback from Mozilla. > Thus, it seems to be a good idea to: > > 1. Henceforth, make NSS only accept certificates by DigiCert with a > maximum validity of 100 days. Let's Encrypt has shown that this is > clearly feasible. > > or > > 2. Henceforth, require DigiCert to revoke a small, randomly (e.g., > using RFC 3797) selected subset of their certificates every day (within 7 days). > If this, e.g., for the same reasons as outlined in these incident > reports, is not possible, it will trigger (a incrementally decreasing > number of) more incident reports. > > Both proposals would lead to more automation and a better > understanding of the requirement of timely revocation, while pushing > the ecosystem in the right direction. For its easiness, the first > proposal would be my favorite but I would be very interested in > hearing other people's thoughts about these proposals. > I don't agree that demanding all certificate customers have "more automation" is desirable. I am very familiar with the Chaos Monkey approach Netflix has implemented and companies like Gremlin that offer similar "Failure as a Service" products, but forcing this on customers seems like a poor idea. Thanks, Peter ___ dev-security-policy mailing list dev-security-policy@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-security-policy smime.p7s Description: S/MIME cryptographic signature ___ dev-security-policy mailing list dev-security-policy@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-security-policy
RE: Underscore characters
The risk is primarily outages of major sites across the web, including certs used in Google wallet. We’re thinking that is a less than desirable result, but we weren’t sure how the Mozilla community would feel/react. We’re still considering revoking all of the certs on Jan 15th based on these discussions. I don’t think we’re asking for leniency (maybe we are if that’s a factor?), but I don’t know what happens if you’re faced with causing outages vs. compliance. I started the conversation because I feel like we should be good netizans and make people aware of what’s going on instead of just following policy. I’m actually surprised at least one other CA that has issued a large number of underscore character certs hasn’t run into the same timing issues. Normally, we would just revoke the certs, but there are a significant number of certs in the Alexa top 100. We’ve told most customers, “No exception”. I also thought it’s better to get the information out there so we can all make rational decisions (DigiCert included) if as many facts are known as possible. We are working with the partners to get the certs revoked before the deadline. Most will. By January 15th, I hope there won’t be too many certs left. Unfortunately, by then it’s also too late to discuss what happens if the cert is not revoked. Ie – what are the benefits of revoking (strict compliance) vs revoking the larger impact certs as they are migrated (incident report). Unfortunately part 2, there’s no guidance on whether an incident report means total distrust v. something on your audit and a stern lecture. I’d happily suffer a lecture than take down a top site. Not so willing to gamble the whole company. This is why we wanted to have the discussion now, despite no violation so far. The response from the browsers is public - that they cannot make that determination. Does that mean we have our answer? Revoke is the only acceptable response? From: James Burton Sent: Thursday, December 27, 2018 2:24 PM To: Ryan Sleevi Cc: Jeremy Rowley ; mozilla-dev-security-policy Subject: Re: Underscore characters On Thu, Dec 27, 2018 at 9:00 PM Ryan Sleevi mailto:r...@sleevi.com> > wrote: I'm not really sure I understand this response at all. I'm hoping you can clarify. On Thu, Dec 27, 2018 at 3:45 PM James Burton mailto:j...@0.me.uk> > wrote: For a CA to intentionally state that they are going to violate the BR requirements means that that CA is under immense pressure to comply with demands or face retribution. I'm not sure I understand how this flows. Comply with whose demands? Face retribution from who, and why? The CA must be under immense pressure to comply with demands from certain customers to determine that they don't have much of a choice but to intentionally violate the BR requirements and by telling community and root stores early they are hoping for leniency. The retribution by them customers could be legal which is outside of this forum but is but it's still relevant to them if that is the case. The severity inflicted on a CA by intentionally violating the BR requirements can be severe. Rolling a dice of chance. Why take the risk? I'm not sure I understand the question at the end, and suspect there's a point to the question I'm missing. The CA is rolling the dice of chance, they are intentionally risking everything by violating the BR requirements and they know that such action can face sanctions or distrust in the wrong case. The question I asked is why are they taking the risk which leads from the first statement. Presumably, a CA stating they're going to violate the BR requirements, knowing the risk to trust that it may pose, would have done everything possible to gather every piece of information so that they could assess the risk of violation is outweighed by whatever other risks (in this case, revocation). If that's the case, is it unreasonable to ask how the CA determined that - which is the root cause analysis question? And how to mitigate whatever other risk (in this case, revocation) poses going forward, so that violating the BRs isn't consistently seen as the "best" option? smime.p7s Description: S/MIME cryptographic signature ___ dev-security-policy mailing list dev-security-policy@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-security-policy
Re: Underscore characters
On Thu, Dec 27, 2018 at 01:19:26PM -0800, Peter Bowen via dev-security-policy wrote: > I don't see how this follows. DigiCert has made it clear they are able to > technically revoke these certificates and presumably are contractually able > to revoke them as well. What is being said is that their customers are > asking them to delay revoking them because the _customers_ have blackout > periods where the customers do not want to make changes to their systems. > DigiCert's customers are saying that they are judging the risk from > revocation is greater than the risk from leaving them unrevoked and asking > DigiCert to not revoke. DigiCert is then presenting this request along to > Mozilla to get feedback from Mozilla. It's worth clarifying that "risk" is not a property of the universe, like magnetic flux density, but rather is assessed relative to specific entities. Thus, when talking about risk, it's worth clearly identifying to whom a risk is associated, as in this variant of part of the above paragraph: > DigiCert's customers are saying that they are judging the risk *to them* > from revocation is greather than the risk *to them* from leaving them > unrevoked I'm sure you're familiar with all this, Peter. I just thought it was worth highlighting for a wider audience, that one entity's assessment of risk to them doesn't make it a physical constant that applies equally to everyone. I find it very helpful when assessing such things to attach explicit markers, somewhat like ensuring I specify both magnitude *and* direction on my vectors. - Matt ___ dev-security-policy mailing list dev-security-policy@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-security-policy
Re: Underscore characters
On Thu, Dec 27, 2018 at 9:00 PM Ryan Sleevi wrote: > I'm not really sure I understand this response at all. I'm hoping you can > clarify. > > On Thu, Dec 27, 2018 at 3:45 PM James Burton wrote: > >> For a CA to intentionally state that they are going to violate the BR >> requirements means that that CA is under immense pressure to comply with >> demands or face retribution. >> > > I'm not sure I understand how this flows. Comply with whose demands? Face > retribution from who, and why? > The CA must be under immense pressure to comply with demands from certain customers to determine that they don't have much of a choice but to intentionally violate the BR requirements and by telling community and root stores early they are hoping for leniency. The retribution by them customers could be legal which is outside of this forum but is but it's still relevant to them if that is the case. > >> The severity inflicted on a CA by intentionally violating the BR >> requirements can be severe. Rolling a dice of chance. Why take the risk? >> > > I'm not sure I understand the question at the end, and suspect there's a > point to the question I'm missing. > The CA is rolling the dice of chance, they are intentionally risking everything by violating the BR requirements and they know that such action can face sanctions or distrust in the wrong case. The question I asked is why are they taking the risk which leads from the first statement. > Presumably, a CA stating they're going to violate the BR requirements, > knowing the risk to trust that it may pose, would have done everything > possible to gather every piece of information so that they could assess the > risk of violation is outweighed by whatever other risks (in this case, > revocation). If that's the case, is it unreasonable to ask how the CA > determined that - which is the root cause analysis question? And how to > mitigate whatever other risk (in this case, revocation) poses going > forward, so that violating the BRs isn't consistently seen as the "best" > option? > ___ dev-security-policy mailing list dev-security-policy@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-security-policy
Re: Underscore characters
On Thu, Dec 27, 2018 at 12:53 PM thomas.gh.horn--- via dev-security-policy < dev-security-policy@lists.mozilla.org> wrote: > > As to why these certificates have to be revoked, you should see this the > other way round: as a very generous service of the community to you and > your customers! > > Certificates with (pseudo-)hostnames in them are clearly invalid, so a > conforming implementation should not accept them for anything and they > should not pose any security risk. Based on this assessment (no revokation > if no security risk), a CA could very well issue a certificate including > any of the (psuedo-)hostnames "example.com_cvs.com", "example.com/cvs.com", > "cvs.com/example.com", "https://example.com/cvs.com;, "example@cvs.com" > to the owner of example.com (who, arguably, has the exact same right to > them as the owner of cvs.com has) and refuse to revoke them. > I'm not clear how you get that the owner of example.com is covered anywhere here. Parsed into labels, these all have com as the label closet to the root and then have 'com_cvs', 'com/cvs', 'com/example', 'com/cvs', and 'com@cvs' as the next label respectively. None have 'example' as the next label. > As to the consequences (in case this really becomes an incident > report/incident reports): this shows a SEVERE lack of ability to revoke > certificates on DigiCert's side, which must have been known AND ACCEPTED > for a long time (this cannot be the first "blackout period" of (in the best > case) 3.5 months). I don't see how this follows. DigiCert has made it clear they are able to technically revoke these certificates and presumably are contractually able to revoke them as well. What is being said is that their customers are asking them to delay revoking them because the _customers_ have blackout periods where the customers do not want to make changes to their systems. DigiCert's customers are saying that they are judging the risk from revocation is greater than the risk from leaving them unrevoked and asking DigiCert to not revoke. DigiCert is then presenting this request along to Mozilla to get feedback from Mozilla. > Thus, it seems to be a good idea to: > > 1. Henceforth, make NSS only accept certificates by DigiCert with a > maximum validity of 100 days. Let's Encrypt has shown that this is clearly > feasible. > > or > > 2. Henceforth, require DigiCert to revoke a small, randomly (e.g., using > RFC 3797) selected subset of their certificates every day (within 7 days). > If this, e.g., for the same reasons as outlined in these incident reports, > is not possible, it will trigger (a incrementally decreasing number of) > more incident reports. > > Both proposals would lead to more automation and a better understanding of > the requirement of timely revocation, while pushing the ecosystem in the > right direction. For its easiness, the first proposal would be my favorite > but I would be very interested in hearing other people's thoughts about > these proposals. > I don't agree that demanding all certificate customers have "more automation" is desirable. I am very familiar with the Chaos Monkey approach Netflix has implemented and companies like Gremlin that offer similar "Failure as a Service" products, but forcing this on customers seems like a poor idea. Thanks, Peter ___ dev-security-policy mailing list dev-security-policy@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-security-policy
Re: Underscore characters
I'm not really sure I understand this response at all. I'm hoping you can clarify. On Thu, Dec 27, 2018 at 3:45 PM James Burton wrote: > For a CA to intentionally state that they are going to violate the BR > requirements means that that CA is under immense pressure to comply with > demands or face retribution. > I'm not sure I understand how this flows. Comply with whose demands? Face retribution from who, and why? > The severity inflicted on a CA by intentionally violating the BR > requirements can be severe. Rolling a dice of chance. Why take the risk? > I'm not sure I understand the question at the end, and suspect there's a point to the question I'm missing. Presumably, a CA stating they're going to violate the BR requirements, knowing the risk to trust that it may pose, would have done everything possible to gather every piece of information so that they could assess the risk of violation is outweighed by whatever other risks (in this case, revocation). If that's the case, is it unreasonable to ask how the CA determined that - which is the root cause analysis question? And how to mitigate whatever other risk (in this case, revocation) poses going forward, so that violating the BRs isn't consistently seen as the "best" option? ___ dev-security-policy mailing list dev-security-policy@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-security-policy
Re: Underscore characters
As to why these certificates have to be revoked, you should see this the other way round: as a very generous service of the community to you and your customers! Certificates with (pseudo-)hostnames in them are clearly invalid, so a conforming implementation should not accept them for anything and they should not pose any security risk. Based on this assessment (no revokation if no security risk), a CA could very well issue a certificate including any of the (psuedo-)hostnames "example.com_cvs.com", "example.com/cvs.com", "cvs.com/example.com", "https://example.com/cvs.com;, "example@cvs.com" to the owner of example.com (who, arguably, has the exact same right to them as the owner of cvs.com has) and refuse to revoke them. As to the consequences (in case this really becomes an incident report/incident reports): this shows a SEVERE lack of ability to revoke certificates on DigiCert's side, which must have been known AND ACCEPTED for a long time (this cannot be the first "blackout period" of (in the best case) 3.5 months). Thus, it seems to be a good idea to: 1. Henceforth, make NSS only accept certificates by DigiCert with a maximum validity of 100 days. Let's Encrypt has shown that this is clearly feasible. or 2. Henceforth, require DigiCert to revoke a small, randomly (e.g., using RFC 3797) selected subset of their certificates every day (within 7 days). If this, e.g., for the same reasons as outlined in these incident reports, is not possible, it will trigger (a incrementally decreasing number of) more incident reports. Both proposals would lead to more automation and a better understanding of the requirement of timely revocation, while pushing the ecosystem in the right direction. For its easiness, the first proposal would be my favorite but I would be very interested in hearing other people's thoughts about these proposals. ___ dev-security-policy mailing list dev-security-policy@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-security-policy
Re: Underscore characters
For a CA to intentionally state that they are going to violate the BR requirements means that that CA is under immense pressure to comply with demands or face retribution. The severity inflicted on a CA by intentionally violating the BR requirements can be severe. Rolling a dice of chance. Why take the risk? On Thu, Dec 27, 2018 at 8:21 PM Ryan Sleevi via dev-security-policy < dev-security-policy@lists.mozilla.org> wrote: > I'm not trying to throw you under the bus here, but I think it's helpful if > you could highlight what new information you see being required, versus > that which is already required. > > I think, yes, you're right that it's not well received if you go violate > the BRs and then, after the fact, say "Hey, yeah, we violated, but here's > why", and finding out that the reasons are met with a lot of skepticism and > the math being shaky, and you can see that from past incident reports it > doesn't go over well. > > But it's also not well received if it's before, and the statement is "Our > customer thinks we should violate the BRs. What would happen if we did, and > what information do you need from us?". That gets into the moral hazard > that Matt spoke to, and is a huge burden on the community where the > expectation is that the CA says "Sorry, we can't do that". > > So the assumption here is that, in all of this discussion, DigiCert's done > everything it can to understand the issue, the timelines, remediation, etc, > and has plans to address both each and every customer and the systemic > issues that have emerged. If that's not the case, then how are we not in > one of those two scenarios above? And if it is the case, isn't that > information readily available by now? > > From the discussions on the incident reports, I feel like that's been the > heart of the questions; which is trying to understand what the root cause > is and what the remediation plan is. The statement "We'll miss the first > deadline, but we'll hit the second", but without any details about how or > why, or the steps being taken to ensure no deadlines are missed in the > future, doesn't really inspire confidence, and is exactly the same kind of > feedback that would be given post-incident. > > On Thu, Dec 27, 2018 at 1:50 PM Jeremy Rowley via dev-security-policy < > dev-security-policy@lists.mozilla.org> wrote: > > > There's a little bit of a "damned if you do, damned if you don't problem > > here". Wait until you have all the information? That's a paddlin'. File > > before you have enough information? That's a paddlin'. I'd appreciate > > better guidance on what Mozilla expects from these incident reports > > timing-wise. > > > > -Original Message- > > From: dev-security-policy > > > On Behalf Of Jeremy Rowley via dev-security-policy > > Sent: Thursday, December 27, 2018 11:47 AM > > To: r...@sleevi.com > > Cc: dev-security-policy@lists.mozilla.org > > Subject: RE: Underscore characters > > > > The original incident report contained all of the details of the initial > > filing. The additional, separated reports are trickling in as I get > enough > > info to post something in reply to the updated questions. As the > questions > > asked have changed from the original 7 in the Mozilla incident report, > > getting the info back takes time. Especially during the holiday season. > > We’re also working to close out as many without an exception as possible. > > Note that the deadline has not passed yet so all of these incident > reports > > are theoretical (and not actually incidents) until Jan 15th. I gave the > > community the total potential number of certificates impacted and the > total > > number of customers so we can have a community discussion on the overall > > risk and get public comments into the process before the deadline passes. > > I’m unaware of any policy at Mozilla or Google that provides guidance on > > how to file expected issues before they happen. If there is, I’d gladly > > follow that. > > > ___ > dev-security-policy mailing list > dev-security-policy@lists.mozilla.org > https://lists.mozilla.org/listinfo/dev-security-policy > ___ dev-security-policy mailing list dev-security-policy@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-security-policy
Re: Underscore characters
I'm not trying to throw you under the bus here, but I think it's helpful if you could highlight what new information you see being required, versus that which is already required. I think, yes, you're right that it's not well received if you go violate the BRs and then, after the fact, say "Hey, yeah, we violated, but here's why", and finding out that the reasons are met with a lot of skepticism and the math being shaky, and you can see that from past incident reports it doesn't go over well. But it's also not well received if it's before, and the statement is "Our customer thinks we should violate the BRs. What would happen if we did, and what information do you need from us?". That gets into the moral hazard that Matt spoke to, and is a huge burden on the community where the expectation is that the CA says "Sorry, we can't do that". So the assumption here is that, in all of this discussion, DigiCert's done everything it can to understand the issue, the timelines, remediation, etc, and has plans to address both each and every customer and the systemic issues that have emerged. If that's not the case, then how are we not in one of those two scenarios above? And if it is the case, isn't that information readily available by now? From the discussions on the incident reports, I feel like that's been the heart of the questions; which is trying to understand what the root cause is and what the remediation plan is. The statement "We'll miss the first deadline, but we'll hit the second", but without any details about how or why, or the steps being taken to ensure no deadlines are missed in the future, doesn't really inspire confidence, and is exactly the same kind of feedback that would be given post-incident. On Thu, Dec 27, 2018 at 1:50 PM Jeremy Rowley via dev-security-policy < dev-security-policy@lists.mozilla.org> wrote: > There's a little bit of a "damned if you do, damned if you don't problem > here". Wait until you have all the information? That's a paddlin'. File > before you have enough information? That's a paddlin'. I'd appreciate > better guidance on what Mozilla expects from these incident reports > timing-wise. > > -Original Message- > From: dev-security-policy > On Behalf Of Jeremy Rowley via dev-security-policy > Sent: Thursday, December 27, 2018 11:47 AM > To: r...@sleevi.com > Cc: dev-security-policy@lists.mozilla.org > Subject: RE: Underscore characters > > The original incident report contained all of the details of the initial > filing. The additional, separated reports are trickling in as I get enough > info to post something in reply to the updated questions. As the questions > asked have changed from the original 7 in the Mozilla incident report, > getting the info back takes time. Especially during the holiday season. > We’re also working to close out as many without an exception as possible. > Note that the deadline has not passed yet so all of these incident reports > are theoretical (and not actually incidents) until Jan 15th. I gave the > community the total potential number of certificates impacted and the total > number of customers so we can have a community discussion on the overall > risk and get public comments into the process before the deadline passes. > I’m unaware of any policy at Mozilla or Google that provides guidance on > how to file expected issues before they happen. If there is, I’d gladly > follow that. > ___ dev-security-policy mailing list dev-security-policy@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-security-policy
Re: Underscore characters
On Wed, Dec 26, 2018 at 1:03 PM Jeremy Rowley wrote: > Much better to treat this question as “We know X is going to happen. > What’s the best way to mitigate the concerns of the community?” Exception > was the wrong word in my original post. I should have used “What would you > like us to do to mitigate when we miss the Jan 15ht deadline?” instead. > Apologies for the confusion there. > As I tried to highlight several times during early discussions, it's not really ideal to have each of these trickle in over time. DigiCert has apparently decided that for 14-15 customers it has sufficient information to know that X is going to happen, based on their risk analysis. Why are we seeing bugs trickle in, such as https://bugzilla.mozilla.org/show_bug.cgi?id=1516545 ? It would seem uncontroversial to suggest that, as part of the risk analysis that DigiCert is claiming has already been done, that it has all the information for an incident report for all of the customers it expects to not revoke certificates for. If it doesn't, then it suggests that the risk analysis is not being done responsibly, and being outsourced to the community to perform. Should we expect another 12 bugs to be filed? If so, when? If not, why? As mentioned, if treating this as part of a "Responding to underscores" incident, then this has the effect of being a slow trickle of an incomplete incident report overall, and incomplete remediation plan, and those tend not to bode well. I don't think it'd really be engaging with mitigating to, say, file a bug on Jan 14th - so how do we move the discussion forward and make sure the facts are available? ___ dev-security-policy mailing list dev-security-policy@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-security-policy
Re: Underscore characters
On Wed, Dec 26, 2018 at 04:13:40PM +, Jeremy Rowley via dev-security-policy wrote: > The trust stores are always free to ignore the CAB Forum mandates and make > their own rules. Mozilla has in the past (see the Mozilla audit > criteria). Whilst the trust stores *can* make their own rules, my observation is that they generally don't do that except as a last resort, or in exigent circumstances. Whilst I wasn't around when it was created, my understanding is that part of the reason the CA/B forum was created was to try and harmonise trust stores' requirements for CAs, so that CAs could work to a single set of requirements. >From that perspective, I can't imagine it would be in the interests of CAs to encourage trust stores to each do their own thing, in general. > The browsers always decide the risk they want to bear and when that risk > becomes unacceptable. The root question is definitely whether this > particular mis-revocation provision would amount to unacceptable risk to > the browsers. Well, *any* amount of risk is "unacceptable" if there is no corresponding reward to be gained from taking that risk. > I don't think we're asking browsers to take on any risk. You disagree with my suggestion that there is a risk that CAs (as a class; I'm not thinking of DigiCert specifically here) will be less likely to engage in a rigorous analysis of the relevant specifications in the future, if not doing so does not result in any harm to them? You don't think there is any risk to trust stores' reputations from the appearance of giving a free pass to CAs which did not follow the rules? There is no risk of appearing to favour certain CAs over others, by effectively punishing those CAs which *did* play by the rules? Let me pose to you a hypothetical. Imagine, for a moment, that DigiCert holds the line, and revokes on January 15. This is certainly going to make your customers annoyed, I certainly get that. But you do the right thing by the web PKI, for which relying parties thank you. Now, one of your competitors, who has all the scruples of an alley tomcat, can easily find out who those companies are -- even before you revoke, just by searching crt.sh for certs issued by DigiCert / Symantec with an underscore in them. They decide to pull a swift one, and put all their more persuasive salespeople on a blitzkrieg campaign to contact those companies and say "hey, I understand DigiCert's done the dirty on you, and they're making you replace all your certificates and rename all your machines. That's gotta be tough. If you switch all your certificate issuance to us, we'll give you two year certs for the existing names, and you don't have to take the risk of renaming everything and breaking your critical systems." There's a decent chance that at least one of your customers would leap at that chance, because whilst it might be risky to change certs, it's a *lot* less risky than changing certs *and* renaming everything, and they have to change their certs anyway. Having lost a customer to a competitor who has gotten that business by offering them something they really should be offering, would you feel a little miffed if Mozilla, when presented with clear evidence that the rules were being violated, decided to give your shady competitor a pass? I fully expect you would be, and you'd be entirely right to feel that way. I'd be right next to you getting miffed, too. Now, do you think there are any CAs out there which have previously issued underscore-containing certificates, who have already told all their customers, flat-out, that they're getting revoked, and they just need to get on with replacing them? Is there any chance that those CAs have lost a customer as a result of that? Do you think there's any chance that they're watching what is happening at the moment *very* closely, and getting ready to be rather miffed if DigiCert gets a pass on the Janusary 15 deadline, when *they* did the hard yards of telling all their customers their certs were getting revoked, and *they* took the business risk of potentially losing customers? > In fact the opposite. The risk of revocation is a browser outage for that > website. The impacts of *that* risk only effect the trust stores to the degree that users may cease to use the associated browser, for another. It has a far greater impact on the organisation, which is as it should be, as it was the organisation's decision to deploy non-conforming names, and an infrastructure that isn't capable of adhering to the agreements that the organisation made with their CA. > A delay in revocation gives the operator for specifically issued > certificates gives them more time to avoid an outage. Thus, risk is > mitigated. Risk is mitigated for one party, (or two), at the cost of increased risk to another party (trust stores). Typically, in a risk transfer transaction, there is some consideration that goes along with agreeing to take on that risk. Of course, if you
Re: Underscore characters
On Wed, Dec 26, 2018 at 06:02:57PM +, Jeremy Rowley via dev-security-policy wrote: > Much better to treat this question as “We know X is going to happen. > What’s the best way to mitigate the concerns of the community?” Exception > was the wrong word in my original post. I should have used “What would > you like us to do to mitigate when we miss the Jan 15ht deadline?” > instead. Apologies for the confusion there. I think that *could* be a productive discussion to have, assuming that (a) it isn't a hypothetical (as in, "tell us what we'd have to do, and we'll decide if that's more pain than just following the rules in the first place") and (b) that there *is* something meaningful that can be done, *before* the deadline would otherwise expire. Insofar as DigiCert has been reasonably upfront about the issues they're facing, and willing to engage in discussion, that isn't a bad thing. Certainly, it's better than the alternative. Apart from that, though, I'm not coming up with anything that *has* to be done before the deadline, that would help to mitigate the problems associated with this incident. One thing that I think *would* be useful would be to know, in sufficient detail that the community can see that it *would* work, how DigiCert is planning to change their practices and processes such that a similar kind of incident cannot happen again in the future. Now that it is absolutely and blindingly obvious that certificates may need to be replaced at any time due to circumstances outside the CA's control, what does DigiCert intend to change in order to ensure that their subscribers will *always* be prepared to replace their certificates at short (ideally, five days, as per the recent BR changes) notice? If DigiCert had a plan to ensure that, and executed on it in a reasonable timeframe, it would go a long way to assuaging my worries, at least, that we're all going to be in this exact same position at some point in the not-too-distant future. - Matt ___ dev-security-policy mailing list dev-security-policy@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-security-policy
Re: Underscore characters
On Wed, Dec 26, 2018 at 1:03 PM Jeremy Rowley wrote: > I don’t think I’m arguing that CAs should ever ignore the BRs. I’m arguing > that deciding the consequences of failing to follow the BRs falls in the > hands of the browsers. But I think you definitely highlighted why this > discussion is confusing. I think all agree on the following: > >1. Failure to revoke by Jan 15th is a non-compliance with the BRs. >2. Non-compliances require an incident report >3. The incident should appear on the audit report. Side note – there >won’t be audit criteria around this particular issue by the time all certs >are revoked. We’re planning to inform our auditor of course (already have >in fact), but without audit criteria any delay in issuance essentially goes >undetected unless someone in this community notices. Because the audit >criteria won’t be updated until well after our audit report, if we were a >bad acting CA, the incident would just never show up. > > > > I think the only thing we disagree on is: > >1. Can the browser say what happens for a failure to comply with the >BRs before the failure happens. > > Right, and I think this is where Matt was getting into the moral hazard side of things, because I think this gets to the heart of the "ignore the BRs". If this is the question being answered, then it should be that every CA who had a customer with some need would, rather than tell that customer no, tell them "Go talk to the browsers". I think that's actively harmful and unacceptable. It's unacceptable, because if shifts the burden wholly on to the browsers to ensure the CAs compliance, and it sets a dangerous precedent that all BRs are up for negotiation, so long as it's before the failure happens. Further, if the CA doesn't like the answer, then they can say no - but all of the cost is borne by the community, in discussing and evaluating, not by the CA, who might decide it's not worth an incident. That's why I posed it as a separate thing - it's not about discussing what happens before the failure happens - but that this specific discussion we're having is about a remediation plan for underscores. This is similar to discussions for remediations for other incidents, such as sub-CAs that aren't following the BRs, metadata in OUs, and other forms of invalid domain names. The 'standard' expectation is 24 hours. SC12 extended that substantially. And we're discussing why some feel that even SC12's proposed remediation plan is problematic, and needing concrete details. > Is that a fair assessment? I see why you wouldn’t want to engage in the > question you asked (“Hypothetically, what would happen if we did (Bad > Thing X)". That would be terrible. Much better to treat this question as > “We know X is going to happen. What’s the best way to mitigate the concerns > of the community?” Exception was the wrong word in my original post. I > should have used “What would you like us to do to mitigate when we miss the > Jan 15ht deadline?” instead. Apologies for the confusion there. > While I think "We know X is going to happen" is still problematic (especially since DigiCert hasn't committed to actually having X happen), I think you're correct that we're discussing about "How do we best remedy this issue in a timely fashion", which is consistent with https://wiki.mozilla.org/CA/Responding_To_An_Incident#Revocation ___ dev-security-policy mailing list dev-security-policy@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-security-policy
RE: Underscore characters
I don’t think I’m arguing that CAs should ever ignore the BRs. I’m arguing that deciding the consequences of failing to follow the BRs falls in the hands of the browsers. But I think you definitely highlighted why this discussion is confusing. I think all agree on the following: 1. Failure to revoke by Jan 15th is a non-compliance with the BRs. 2. Non-compliances require an incident report 3. The incident should appear on the audit report. Side note – there won’t be audit criteria around this particular issue by the time all certs are revoked. We’re planning to inform our auditor of course (already have in fact), but without audit criteria any delay in issuance essentially goes undetected unless someone in this community notices. Because the audit criteria won’t be updated until well after our audit report, if we were a bad acting CA, the incident would just never show up. I think the only thing we disagree on is: 4. Can the browser say what happens for a failure to comply with the BRs before the failure happens. Is that a fair assessment? I see why you wouldn’t want to engage in the question you asked (“Hypothetically, what would happen if we did (Bad Thing X)". That would be terrible. Much better to treat this question as “We know X is going to happen. What’s the best way to mitigate the concerns of the community?” Exception was the wrong word in my original post. I should have used “What would you like us to do to mitigate when we miss the Jan 15ht deadline?” instead. Apologies for the confusion there. Jeremy From: Ryan Sleevi Sent: Wednesday, December 26, 2018 10:00 AM To: Jeremy Rowley Cc: dev-security-policy@lists.mozilla.org Subject: Re: Underscore characters On Wed, Dec 26, 2018 at 11:13 AM Jeremy Rowley via dev-security-policy mailto:dev-security-policy@lists.mozilla.org> > wrote: Hey Matt, The trust stores are always free to ignore the CAB Forum mandates and make their own rules. Mozilla has in the past (see the Mozilla audit criteria exception for other audits outside of Webtrust and ETSI). The root stores are also the entities that determine what happens if the rules are violated. Thus, we're asking what the violation of this revocation timeline results in and whether Mozilla is enforcing the CAB Forum requirement. The browsers always decide the risk they want to bear and when that risk becomes unacceptable. The question we're asking is whether this particular mis-revocation provision would amount to unacceptable risk to the browsers. I don't think we're asking browsers to take on any risk. In fact the opposite. The risk of revocation is a browser outage for that website. A delay in revocation gives the operator for specifically issued certificates gives them more time to avoid an outage. Thus, risk is mitigated. A poor explanation, but I think we have to identify what the risk is before browsers can say they are taking on anything additional. The "CAs may be doing bad things in the dark" allegation can't be responded to because it's too vague. I'm also troubled to think that might be a concern as our policy is to over-report issues. Plus, that risk is pretty hard to sell to management as an immediate threat requiring replacement of their certificates. This lack of definition on the problem is also the main difference between this event and a compromised key. Explaining key compromise to executive management for an emergency exception to a blackout period is a lot different than explaining why hundreds of certificates require replacement because they contain underscores. I think everyone would benefit (myself included) if I could get more information about why underscore characters themselves present an actual risk. If we could get a statement on that, you'd see a lot less confusion. Jeremy Jeremy, While I can't speak for Wayne, I tried to highlight how dangerous and problematic this thinking is and framing. By framing it as you have, it makes it much more difficult to see this as a productive discussion about handling revocation following an incident, and instead about a CA arguing that they should be able to ignore the BRs at will. I'm sure you can see how that latter framing is especially problematic, and I think arguments that try to present it as such have a chance at steering the conversation very negatively. You've heard from two browsers at least (Mozilla and Google) that they expect an incident report, which means that they are enforcing the Baseline Requirements and do view this as non-compliance by a CA. There is no exception being granted - it's non-compliance. Further, the discussion you're looking to have is seemingly not about whether this particular incident is problematic in-and-of-itself, even though you've framed it as such here, but instead whether the pattern and set of incidents represents a concern about the ongoing ri
Re: Underscore characters
On Wed, Dec 26, 2018 at 11:13 AM Jeremy Rowley via dev-security-policy < dev-security-policy@lists.mozilla.org> wrote: > Hey Matt, > > The trust stores are always free to ignore the CAB Forum mandates and make > their own rules. Mozilla has in the past (see the Mozilla audit criteria > exception for other audits outside of Webtrust and ETSI). The root stores > are also the entities that determine what happens if the rules are > violated. Thus, we're asking what the violation of this revocation timeline > results in and whether Mozilla is enforcing the CAB Forum requirement. The > browsers always decide the risk they want to bear and when that risk > becomes unacceptable. The question we're asking is whether this particular > mis-revocation provision would amount to unacceptable risk to the > browsers. > > I don't think we're asking browsers to take on any risk. In fact the > opposite. The risk of revocation is a browser outage for that website. A > delay in revocation gives the operator for specifically issued certificates > gives them more time to avoid an outage. Thus, risk is mitigated. A poor > explanation, but I think we have to identify what the risk is before > browsers can say they are taking on anything additional. The "CAs may be > doing bad things in the dark" allegation can't be responded to because it's > too vague. I'm also troubled to think that might be a concern as our policy > is to over-report issues. Plus, that risk is pretty hard to sell to > management as an immediate threat requiring replacement of their > certificates. This lack of definition on the problem is also the main > difference between this event and a compromised key. Explaining key > compromise to executive management for an emergency exception to a blackout > period is a lot different than explaining why hundreds of certificates > require replacement because they contain underscores. I think everyone > would benefit (myself included) if I could get more information about why > underscore characters themselves present an actual risk. If we could get a > statement on that, you'd see a lot less confusion. > > Jeremy > Jeremy, While I can't speak for Wayne, I tried to highlight how dangerous and problematic this thinking is and framing. By framing it as you have, it makes it much more difficult to see this as a productive discussion about handling revocation following an incident, and instead about a CA arguing that they should be able to ignore the BRs at will. I'm sure you can see how that latter framing is especially problematic, and I think arguments that try to present it as such have a chance at steering the conversation very negatively. You've heard from two browsers at least (Mozilla and Google) that they expect an incident report, which means that they are enforcing the Baseline Requirements and do view this as non-compliance by a CA. There is no exception being granted - it's non-compliance. Further, the discussion you're looking to have is seemingly not about whether this particular incident is problematic in-and-of-itself, even though you've framed it as such here, but instead whether the pattern and set of incidents represents a concern about the ongoing risk posed by continued trust. A poor analogy, but one that hopefully highlights the flaws in the argument you're making, is a bit like asking "What's so bad about stealing a candy bar from the shop", while trying to ignore whether you robbed the till the day previous or have been stealing every day the past week. The framing that seems to have resonated is that we are NOT talking about whether or not stealing candy bars is OK and acceptable. We've seemingly agreed it's bad, and thus (in the CA space) are expecting an incident report and treating it as an incident. It would be extremely risky to suggest that stealing is sometimes OK, both in the immediate and long-term. The question being discussed is what to do if (or, in this case, when) you're caught stealing, and what it would look like. Matt's moral hazard is absolutely correct with respect to legitimizing things - especially treating them as non-incidents. Similarly, I have concerns with the ideas that CAs can or should ask the community "Hypothetically, what would happen if we did (Bad Thing X)" - I think that demonstrates less than stellar trust. That's why I suggested that this is a continuation of the discussion about underscores - "So, a CA did bad thing X, how do we get the ecosystem whole without causing unnecessary challenges" - rather than being on trying to segment out the hierarchy into compromise vs CA negligence. ___ dev-security-policy mailing list dev-security-policy@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-security-policy
RE: Underscore characters
Hey Matt, The trust stores are always free to ignore the CAB Forum mandates and make their own rules. Mozilla has in the past (see the Mozilla audit criteria exception for other audits outside of Webtrust and ETSI). The root stores are also the entities that determine what happens if the rules are violated. Thus, we're asking what the violation of this revocation timeline results in and whether Mozilla is enforcing the CAB Forum requirement. The browsers always decide the risk they want to bear and when that risk becomes unacceptable. The question we're asking is whether this particular mis-revocation provision would amount to unacceptable risk to the browsers. I don't think we're asking browsers to take on any risk. In fact the opposite. The risk of revocation is a browser outage for that website. A delay in revocation gives the operator for specifically issued certificates gives them more time to avoid an outage. Thus, risk is mitigated. A poor explanation, but I think we have to identify what the risk is before browsers can say they are taking on anything additional. The "CAs may be doing bad things in the dark" allegation can't be responded to because it's too vague. I'm also troubled to think that might be a concern as our policy is to over-report issues. Plus, that risk is pretty hard to sell to management as an immediate threat requiring replacement of their certificates. This lack of definition on the problem is also the main difference between this event and a compromised key. Explaining key compromise to executive management for an emergency exception to a blackout period is a lot different than explaining why hundreds of certificates require replacement because they contain underscores. I think everyone would benefit (myself included) if I could get more information about why underscore characters themselves present an actual risk. If we could get a statement on that, you'd see a lot less confusion. Jeremy -Original Message- From: dev-security-policy On Behalf Of Matt Palmer via dev-security-policy Sent: Thursday, December 20, 2018 4:54 PM To: dev-security-policy@lists.mozilla.org Subject: Re: Underscore characters On Thu, Dec 20, 2018 at 10:34:21PM +, Jeremy Rowley via dev-security-policy wrote: > Here’s the first of the companies. Figured I’d do one and see if it has the > information you want. > > https://clicktime.symantec.com/a/1/Vso73rFeURLZ94HBeuxBLdmk1isdwCRJ1YP > CMFAxstM=?d=I3ucqXub-yr0TW9Ocbs-YTBkM2F0beNtptvGgqlq3YEDH6Fzq26eV5Vign > YLpVcHu_P3Gdnnz-qiPqcKis3N25Fp-2RGfSxyMcVFVUNXL4_EQlFrw0BYTZpuPCQdk5mm > -nSlrDH6uc4OgNw1QYDQACt6RPMqV8qWIioLa1QehqMa3nJlcGcR8b3abEqcOYnxwAZBxE > lxsBIDqHumeVxhaczrPgjNCOobWmoaqVYwIp9ZGyEADoOrpFVhL_p7uYKkSi1JVOAePuQg > WB8Xu_QHkdm22N_ZkxRxbBdD1Jc0xy4YuXr58Tfv96bX0LGUeM69JWT8_jRwCLwOPgMJXW > pRNZec6GRDumz3V2itO4ujx1MRsegZuKhuwUOxc3M0QrEHr734ym37mw%3D%3D=https > %3A%2F%2Fbugzilla.mozilla.org%2Fshow_bug.cgi%3Fid%3D1515788 Complete side-note: when the customer said you couldn't identify them by name, did they know you were going to be linking to a whole pile of certificates with their organizationName in them? > I think this answers all of your questions (except Ryan’s question > about remediation). Could you let me know if more detail is required > or if you’d like additional info included? The question that comes to my mind most of all is tangentially related to Ryan's question around long-term remediation, but I'll ask it anyway as it's at least somewhat independent. You state in the bug you linked that "[The certificates] can't be replaced before [April 30, 2019] because of the code freeze and the risk of an outage". That is an absolute statement, which I take to mean that there is *no* possibility of certificates being replaced in the freeze period (Oct 15-Feb 1). So, my question is: what would this organization do if they had suffered a key compromise (to take one possible reason for immediate revocation, which happens to be forefront in my mind) on Oct 16? Would this organization continue to use a certificate which uses a known-compromised key until possibly as late as April 30 -- which, if my counting-on-fingers is correct, is approximately six and a half months? Would DigiCert consider not revoking the certificate with a compromised key for that long, if the customer asked them to? Would DigiCert expect trust stores to bless that decision? If the answer to the above questions is "no, of course not!" (as I would sincerely hope they would be), then your absolute statement that the certificates "*can't* be replaced" should probably read something more like "the organization would prefer not to replace the certificates before then, because it is a lot of work and entails some degree of risk". Which takes us back to the calculus of options: 1. DigiCert revokes, organizati
Re: Statement on the Sunset of Underscore Characters
On Fri, Dec 21, 2018 at 4:43 PM Jeremy Rowley wrote: > But this part isn't true "Browsers are not capable of granting > 'exceptions' to the Baseline Requirements", at least for Mozilla. See the > Mozilla auditor requirements for example. Perhaps better stated that they > don't have to implement the standards they don't like? > I mean, we can get pedantic if you want to get pedantic, but I don't think it really conflicts with any of the discussion here. I'll use WebTrust as an example here: - When a CA engages with an auditor, they specify the criteria to be used (i.e. http://www.webtrust.org/principles-and-criteria/item83172.aspx ). - If we assume a US-based auditor, then we're talking an auditor bound to AICPA with regards to attestation engagements against those criteria - The auditor may consider the perspective of Google and Mozilla in some regards, but those are secondary sources for them, and the auditor has professional obligations captured within AICPA's various codes that express what their expectations and interpretations are, as well as what they communicate, and no browser can override those - As a result, we can't tell you "You won't get a qualified audit" on that, because that's up to the auditor. Now, it's true that there are ways to change that calculus, by redefining the audit expectations - For example, if Mozilla or Google engaged the auditor, to audit a third-party (the CA), then Mozilla or Google could develop the criteria for that assessment in collaboration with the auditor (Agreed Upon Procedures), for which the auditor would then report on - But then you're no longer using those WebTrust criteria (or you may, with modified interpretations), and I'm sure if Don or Jeff from the WebTrust TF are lurking here, they'd probably chime in with a whole host of complications Now, the point of a CA getting audited is so that they can submit it to the Root Program that expects such audits. Such Root Programs can define whatever expectations they want - such as allowing anyone to perform the audit - with whatever the risks or tradeoffs that may come with. In the case of most root programs, you already see the acceptance of two audit criteria (WebTrust and ETSI), with sets of licensed auditors managed independently by those programs. Mozilla's policy, as the example you called out, specifically allows it to extend or reduce the set of auditors, but without respect to the criteria. Certainly, a root program can, upon receipt of a qualified audit, determine that no further action is warranted, or that the qualification is not a concern. But that does not absolve the auditor of their own duties (modulo if Mozilla were to appoint a random auditor or to directly negotiate the criteria, procedures, or reporting) The point being is that, short of directly developing the audit criteria or engaging directly with the auditor with respect to third-party audits using agreed-upon procedures, Root Programs - such as those operated by Google or Mozilla - don't really say whether or not something is or is not a qualification/non-conformity. That's the auditor, using fixed criteria, expressing their opinion based on the material facts. We do consider the auditor's opinion and expression of those facts, and they are relevant to Root Programs, but they don't somehow cause the auditor to say "Oh, my opinion doesn't matter, I'll do whatever they say" (Unless, again, that's what the engagement agreed to) This is all pedantry lost in the woods, because I don't think any CA can or should reasonably expect that the view of one browser (which is merely a consumer of the report, not the initiator) could declare something not-a-qualification. They might declare it not-relevant for their program, but that's an entirely different thing - that's part of the incident response process I described, which is fairly consistent among all programs. ___ dev-security-policy mailing list dev-security-policy@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-security-policy
RE: Statement on the Sunset of Underscore Characters
But this part isn't true "Browsers are not capable of granting 'exceptions' to the Baseline Requirements", at least for Mozilla. See the Mozilla auditor requirements for example. Perhaps better stated that they don't have to implement the standards they don't like? -Original Message- From: dev-security-policy On Behalf Of Ryan Sleevi via dev-security-policy Sent: Friday, December 21, 2018 2:22 PM To: Wayne Thayer Cc: mozilla-dev-security-policy Subject: Re: Statement on the Sunset of Underscore Characters On Fri, Dec 21, 2018 at 1:54 PM Wayne Thayer via dev-security-policy < dev-security-policy@lists.mozilla.org> wrote: > Since a number of questions and concerns have been raised regarding > the sunset of underscore characters in dNSNames, I would like to > summarize Mozilla’s position on the issue as follows: > > In early November, the CA/Browser Forum passed ballot SC12 [1], > creating a sunset period aimed at ending the practice of placing > underscore characters > (“_”) in the subjectAlternativeName (SAN) attribute of > publicly-trusted TLS certificates. The ballot requires revocation of > most existing certificates with underscore characters in SANs before > 15-Jan 2019, but allows continued use of these certificates through > April 2019 if they are valid for less than 30 days. > > A thorough review of RFC 5280 and its references shows that underscore > characters have never been permitted in dNSNames in the SAN of > compliant certificates. However, for many years this was a commonly > accepted practice, and all major browsers currently ignore this requirement. > > In 2017, a ballot that attempted (among other things) to create an > exception to RFC 5280 for these certificates [2] in the CA/Browser > Forum Baseline Requirements was rejected [3], in-part due to > objections to this exception. At that time, some CAs decided to stop > issuing these certificates, while others continued. When the situation > resurfaced earlier this year, there was spirited debate about how best > to resolve the problem, and the compromise defined in ballot SC12 > eventually emerged as the most viable approach. > > Mozilla has been asked by users of these certificates - primarily in > the financial services industry - and their CAs to make exceptions to > extend the revocation deadline until their systems can be updated. We > don’t have the power to unilaterally change the Baseline Requirements > (only the CA/Browser Forum can do that), but Mozilla can take a > position on our plans to enforce them. However, simply granting CAs a > “free pass” to ignore the revocation requirement is, we believe, not > in the best interest of our users, fundamentally because it sets the > expectation that the Baseline Requirements are negotiable. It is also: > * unfair to CAs who voted in favor of ballot SC12 under the assumption > that it would be enforced > * unfair to the CAs who avoided this situation by ending this practice > last year > * unfair to the CAs (and their impacted customers) who will revoke all > these certificates on time > > This doesn’t mean that we are insensitive to the potential disruption > caused by the revocation of this type of certificate. > > We sincerely hope that affected organizations make every effort to > meet the revocation deadline. If that is not possible for each and > every certificate containing an underscore in the SAN, our expectation > is for CAs to treat any failure to adhere to ballot SC12 as a policy > violation. Should a CA choose not to revoke certificates with > underscores in a SAN prior to 15-January 2019, we expect the > individual circumstances that led to the decision for each Subscriber > organization to be provided in a detailed incident report.[4] For this > situation, we prefer for the incident report to be submitted > immediately so that the Mozilla community can make our best effort to > evaluate the information and identify any gaps or unmet expectations > before the 15-January revocation deadline. We believe that this > approach creates the best incentives to balance the various risks and > to maximize disclosure, and in so doing helps us to improve the > trustworthiness of the web PKI. > (In a Chrome capacity) Thanks for sharing this, Wayne. On behalf of Chrome, we want to echo our support for this statement and the expectations, and believe it is consistent with our expectations on the matter. To emphasize a few specific points here: Browsers are not capable of granting "exceptions" to the Baseline Requirements. We, Chrome, expect any violations identified by CA management or their auditors to be disclosed within their audit reports, management letters, and to the broader community, the principles
Re: Statement on the Sunset of Underscore Characters
On Fri, Dec 21, 2018 at 1:54 PM Wayne Thayer via dev-security-policy < dev-security-policy@lists.mozilla.org> wrote: > Since a number of questions and concerns have been raised regarding the > sunset of underscore characters in dNSNames, I would like to summarize > Mozilla’s position on the issue as follows: > > In early November, the CA/Browser Forum passed ballot SC12 [1], creating a > sunset period aimed at ending the practice of placing underscore characters > (“_”) in the subjectAlternativeName (SAN) attribute of publicly-trusted TLS > certificates. The ballot requires revocation of most existing certificates > with underscore characters in SANs before 15-Jan 2019, but allows continued > use of these certificates through April 2019 if they are valid for less > than 30 days. > > A thorough review of RFC 5280 and its references shows that underscore > characters have never been permitted in dNSNames in the SAN of compliant > certificates. However, for many years this was a commonly accepted > practice, and all major browsers currently ignore this requirement. > > In 2017, a ballot that attempted (among other things) to create an > exception to RFC 5280 for these certificates [2] in the CA/Browser Forum > Baseline Requirements was rejected [3], in-part due to objections to this > exception. At that time, some CAs decided to stop issuing these > certificates, while others continued. When the situation resurfaced earlier > this year, there was spirited debate about how best to resolve the problem, > and the compromise defined in ballot SC12 eventually emerged as the most > viable approach. > > Mozilla has been asked by users of these certificates - primarily in the > financial services industry - and their CAs to make exceptions to extend > the revocation deadline until their systems can be updated. We don’t have > the power to unilaterally change the Baseline Requirements (only the > CA/Browser Forum can do that), but Mozilla can take a position on our plans > to enforce them. However, simply granting CAs a “free pass” to ignore the > revocation requirement is, we believe, not in the best interest of our > users, fundamentally because it sets the expectation that the Baseline > Requirements are negotiable. It is also: > * unfair to CAs who voted in favor of ballot SC12 under the assumption that > it would be enforced > * unfair to the CAs who avoided this situation by ending this practice last > year > * unfair to the CAs (and their impacted customers) who will revoke all > these certificates on time > > This doesn’t mean that we are insensitive to the potential disruption > caused by the revocation of this type of certificate. > > We sincerely hope that affected organizations make every effort to meet the > revocation deadline. If that is not possible for each and every certificate > containing an underscore in the SAN, our expectation is for CAs to treat > any failure to adhere to ballot SC12 as a policy violation. Should a CA > choose not to revoke certificates with underscores in a SAN prior to > 15-January 2019, we expect the individual circumstances that led to the > decision for each Subscriber organization to be provided in a detailed > incident report.[4] For this situation, we prefer for the incident report > to be submitted immediately so that the Mozilla community can make our best > effort to evaluate the information and identify any gaps or unmet > expectations before the 15-January revocation deadline. We believe that > this approach creates the best incentives to balance the various risks and > to maximize disclosure, and in so doing helps us to improve the > trustworthiness of the web PKI. > (In a Chrome capacity) Thanks for sharing this, Wayne. On behalf of Chrome, we want to echo our support for this statement and the expectations, and believe it is consistent with our expectations on the matter. To emphasize a few specific points here: Browsers are not capable of granting "exceptions" to the Baseline Requirements. We, Chrome, expect any violations identified by CA management or their auditors to be disclosed within their audit reports, management letters, and to the broader community, the principles of which are captured in our Root Certificate Policy [A]. We expect there to be a public incident report and discussion, to ensure that the information necessary to ensure that the scope of the issue has been identified, the cause of the issues, and that the plans for remediation are consistently executed upon and meaningfully address the causes. In all of this, we welcome public participation and transparency, and expect the CA, which is ultimately responsible for any incident, to be the primary party engaging on the incident. Similarly, we expect any information expected to be considered
Statement on the Sunset of Underscore Characters
Since a number of questions and concerns have been raised regarding the sunset of underscore characters in dNSNames, I would like to summarize Mozilla’s position on the issue as follows: In early November, the CA/Browser Forum passed ballot SC12 [1], creating a sunset period aimed at ending the practice of placing underscore characters (“_”) in the subjectAlternativeName (SAN) attribute of publicly-trusted TLS certificates. The ballot requires revocation of most existing certificates with underscore characters in SANs before 15-Jan 2019, but allows continued use of these certificates through April 2019 if they are valid for less than 30 days. A thorough review of RFC 5280 and its references shows that underscore characters have never been permitted in dNSNames in the SAN of compliant certificates. However, for many years this was a commonly accepted practice, and all major browsers currently ignore this requirement. In 2017, a ballot that attempted (among other things) to create an exception to RFC 5280 for these certificates [2] in the CA/Browser Forum Baseline Requirements was rejected [3], in-part due to objections to this exception. At that time, some CAs decided to stop issuing these certificates, while others continued. When the situation resurfaced earlier this year, there was spirited debate about how best to resolve the problem, and the compromise defined in ballot SC12 eventually emerged as the most viable approach. Mozilla has been asked by users of these certificates - primarily in the financial services industry - and their CAs to make exceptions to extend the revocation deadline until their systems can be updated. We don’t have the power to unilaterally change the Baseline Requirements (only the CA/Browser Forum can do that), but Mozilla can take a position on our plans to enforce them. However, simply granting CAs a “free pass” to ignore the revocation requirement is, we believe, not in the best interest of our users, fundamentally because it sets the expectation that the Baseline Requirements are negotiable. It is also: * unfair to CAs who voted in favor of ballot SC12 under the assumption that it would be enforced * unfair to the CAs who avoided this situation by ending this practice last year * unfair to the CAs (and their impacted customers) who will revoke all these certificates on time This doesn’t mean that we are insensitive to the potential disruption caused by the revocation of this type of certificate. We sincerely hope that affected organizations make every effort to meet the revocation deadline. If that is not possible for each and every certificate containing an underscore in the SAN, our expectation is for CAs to treat any failure to adhere to ballot SC12 as a policy violation. Should a CA choose not to revoke certificates with underscores in a SAN prior to 15-January 2019, we expect the individual circumstances that led to the decision for each Subscriber organization to be provided in a detailed incident report.[4] For this situation, we prefer for the incident report to be submitted immediately so that the Mozilla community can make our best effort to evaluate the information and identify any gaps or unmet expectations before the 15-January revocation deadline. We believe that this approach creates the best incentives to balance the various risks and to maximize disclosure, and in so doing helps us to improve the trustworthiness of the web PKI. - Wayne [1] https://cabforum.org/2018/11/12/ballot-sc-12-sunset-of-underscores-in-dnsnames/ [2] https://cabforum.org/pipermail/public/2017-July/011653.html [3] https://cabforum.org/pipermail/public/2017-July/011708.html [4] https://wiki.mozilla.org/CA/Responding_To_An_Incident#Incident_Report ___ dev-security-policy mailing list dev-security-policy@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-security-policy
Re: Underscore characters
On Thu, Dec 20, 2018 at 4:54 PM Matt Palmer via dev-security-policy < dev-security-policy@lists.mozilla.org> wrote: > On Thu, Dec 20, 2018 at 10:34:21PM +, Jeremy Rowley via > dev-security-policy wrote: > > Here’s the first of the companies. Figured I’d do one and see if it has > the information you want. > > > > https://bugzilla.mozilla.org/show_bug.cgi?id=1515788 > > Complete side-note: when the customer said you couldn't identify them by > name, did they know you were going to be linking to a whole pile of > certificates with their organizationName in them? > > > I think this answers all of your questions (except Ryan’s question about > > remediation). Could you let me know if more detail is required or if > > you’d like additional info included? > > The question that comes to my mind most of all is tangentially related to > Ryan's question around long-term remediation, but I'll ask it anyway as > it's > at least somewhat independent. > > You state in the bug you linked that "[The certificates] can't be replaced > before [April 30, 2019] because of the code freeze and the risk of an > outage". That is an absolute statement, which I take to mean that there is > *no* possibility of certificates being replaced in the freeze period (Oct > 15-Feb 1). > > So, my question is: what would this organization do if they had suffered a > key compromise (to take one possible reason for immediate revocation, which > happens to be forefront in my mind) on Oct 16? Would this organization > continue to use a certificate which uses a known-compromised key until > possibly as late as April 30 -- which, if my counting-on-fingers is > correct, > is approximately six and a half months? Would DigiCert consider not > revoking the certificate with a compromised key for that long, if the > customer asked them to? Would DigiCert expect trust stores to bless that > decision? > > I agree that more information is needed here. My hypothetical is that of a critical vulnerability in one of Organization One's systems being discovered on 16-Oct. Does Organization One hold off on patching until Feb? If not, what makes these certificates different? Why is so much coordination required if they are just used in browsers? Was a risk assessment performed to evaluate the possibility of replacing them during the freeze? Are routine changes permitted during the change? If so, why is a certificate replacement not a routine change? If the answer to the above questions is "no, of course not!" (as I would > sincerely hope they would be), then your absolute statement that the > certificates "*can't* be replaced" should probably read something more like > "the organization would prefer not to replace the certificates before then, > because it is a lot of work and entails some degree of risk". Which takes > us back to the calculus of options: > > 1. DigiCert revokes, organization changes domain names and certs: the >organization does lots of work, and takes the risk of Things Going Very >Wrong because of domain name and cert changes. > > 2. DigiCert revokes, organization switches to 30 day certs: the > organization >does lots of work, and takes the risk of Things Going Very Wrong because > of cert changes. > > 3. DigiCert revokes, organization doesn't swap certs: the organization does >lots of work, and takes the risk of Things Going Very Wrong because of >revocation checking. > > 4. DigiCert doesn't revoke, on its own behest: the organization does no >work, takes no risk, while DigiCert takes the risk of consequences from >trust stores of failing to follow the BRs. > > 5. DigiCert doesn't revoke, trust stores bless this decision: the >organization does no work and takes no risk, DigiCert takes no risk, >while trust stores take the risk that CAs' future behaviour is > influenced >by the appearance that deliberately failing to adhere to the BRs carries >little-to-no consequences. > > Given that the entities which made the decisions which led to the current > situation are DigiCert (for allowing issuance of invalid certificates) and > this organization (which failed to heed their subscriber agreement and > decided to build an infrastructure which cannot be adjusted on the > timeframe > required under their subscriber agreement), can you explain why it is > reasonable for the trust stores -- which appear to have done nothing > inappropriate to cause the current state of affairs -- to be the ones > taking > on *any* of the risk here? > > Please don't think that I'm not sympathetic to the situation this "Major > Pharmacy Benefits Manager" and DigiCert are in -- my day job is keeping > production systems up and running, and the idea of having to make fast > changes isn't one I enjoy. Running a commercial entity is never made any > easier when you have to cause problems for your customers. SC12 is not the > phase-out plan *I* would have chosen, were I Benevolent Dictator of the > Internet. However,
Re: Underscore characters
On Thu, Dec 20, 2018 at 10:34:21PM +, Jeremy Rowley via dev-security-policy wrote: > Here’s the first of the companies. Figured I’d do one and see if it has the > information you want. > > https://bugzilla.mozilla.org/show_bug.cgi?id=1515788 Complete side-note: when the customer said you couldn't identify them by name, did they know you were going to be linking to a whole pile of certificates with their organizationName in them? > I think this answers all of your questions (except Ryan’s question about > remediation). Could you let me know if more detail is required or if > you’d like additional info included? The question that comes to my mind most of all is tangentially related to Ryan's question around long-term remediation, but I'll ask it anyway as it's at least somewhat independent. You state in the bug you linked that "[The certificates] can't be replaced before [April 30, 2019] because of the code freeze and the risk of an outage". That is an absolute statement, which I take to mean that there is *no* possibility of certificates being replaced in the freeze period (Oct 15-Feb 1). So, my question is: what would this organization do if they had suffered a key compromise (to take one possible reason for immediate revocation, which happens to be forefront in my mind) on Oct 16? Would this organization continue to use a certificate which uses a known-compromised key until possibly as late as April 30 -- which, if my counting-on-fingers is correct, is approximately six and a half months? Would DigiCert consider not revoking the certificate with a compromised key for that long, if the customer asked them to? Would DigiCert expect trust stores to bless that decision? If the answer to the above questions is "no, of course not!" (as I would sincerely hope they would be), then your absolute statement that the certificates "*can't* be replaced" should probably read something more like "the organization would prefer not to replace the certificates before then, because it is a lot of work and entails some degree of risk". Which takes us back to the calculus of options: 1. DigiCert revokes, organization changes domain names and certs: the organization does lots of work, and takes the risk of Things Going Very Wrong because of domain name and cert changes. 2. DigiCert revokes, organization switches to 30 day certs: the organization does lots of work, and takes the risk of Things Going Very Wrong because of cert changes. 3. DigiCert revokes, organization doesn't swap certs: the organization does lots of work, and takes the risk of Things Going Very Wrong because of revocation checking. 4. DigiCert doesn't revoke, on its own behest: the organization does no work, takes no risk, while DigiCert takes the risk of consequences from trust stores of failing to follow the BRs. 5. DigiCert doesn't revoke, trust stores bless this decision: the organization does no work and takes no risk, DigiCert takes no risk, while trust stores take the risk that CAs' future behaviour is influenced by the appearance that deliberately failing to adhere to the BRs carries little-to-no consequences. Given that the entities which made the decisions which led to the current situation are DigiCert (for allowing issuance of invalid certificates) and this organization (which failed to heed their subscriber agreement and decided to build an infrastructure which cannot be adjusted on the timeframe required under their subscriber agreement), can you explain why it is reasonable for the trust stores -- which appear to have done nothing inappropriate to cause the current state of affairs -- to be the ones taking on *any* of the risk here? Please don't think that I'm not sympathetic to the situation this "Major Pharmacy Benefits Manager" and DigiCert are in -- my day job is keeping production systems up and running, and the idea of having to make fast changes isn't one I enjoy. Running a commercial entity is never made any easier when you have to cause problems for your customers. SC12 is not the phase-out plan *I* would have chosen, were I Benevolent Dictator of the Internet. However, now that it is the plan which has been put in place, following the rules of the game trust stores and CAs have agreed to play by, I find it disconcerting that DigiCert and Organization One want to shift the risk for their decisions onto trust stores. What is the *benefit* to trust stores in taking on that risk, to the undeniable benefit to DigiCert and Organization One? Perhaps, at the end of the day, *that* is the real question to be answered. - Matt ___ dev-security-policy mailing list dev-security-policy@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-security-policy
RE: Underscore characters
Hey all, Here’s the first of the companies. Figured I’d do one and see if it has the information you want. https://bugzilla.mozilla.org/show_bug.cgi?id=1515788 I think this answers all of your questions (except Ryan’s question about remediation). Could you let me know if more detail is required or if you’d like additional info included? Thanks! Jeremy From: Wayne Thayer Sent: Thursday, December 20, 2018 12:25 PM To: Ryan Sleevi Cc: Jeremy Rowley ; mozilla-dev-security-policy Subject: Re: Underscore characters Jeremy, It's good to hear that you do believe you can provide the necessary level of information prior to 15-Jan. Given that, I'm now thinking of this as if it were a normal incident except that we're moving the reporting prior to the incident actually occurring. With 15 affected customers, and perhaps many more deployment scenarios, I would ask you to break this into separate incident reports per customer as Ryan has previously suggested/requested. I can understand the desire to not have 15 separate compliance bugs filed under DigiCert for what is arguably the same issue, but I think that reporting separately per customer will help to ensure that we receive the level of detail needed to assess the hypothetical incident. I'm also not a fan of the proposed 30-April exceptional revocation deadline. This provides zero opportunity to employ the 30-day cert option if something is missed, and it seems to be an arbitrary date rather than an evaluation of the earliest date by which each customer can safely replace their non-compliant certificates. - Wayne On Thu, Dec 20, 2018 at 11:16 AM Ryan Sleevi via dev-security-policy mailto:dev-security-policy@lists.mozilla.org> > wrote: Thanks for filing this, Jeremy. If I understand correctly, the request DigiCert is asking is: "If we submitted this as an incident report, would it be likely that conversations about distrusting DigiCert would begin?", and that's what you're trying to gauge from the community? I think Wayne's already captured the "We need more information", but I think it may be helpful to explain the reasoning and thinking here. The Baseline Requirements and Root Program policies exist for a purpose: To provide a consistent set of expectations for CAs, which meet the security needs of the products using or operating those policies. As these policies tend to call out, a CA may be removed (distrusted) for any reason or no reason at all - it's entirely at the Program discretion. That said, history tends to largely see removals for patterns of issues that, in aggregate, demonstrate an ongoing and significant risk to users and the Internet at large, although there have been CAs removed for single incidents in the past - such as key compromise or issuing MITM certificates. As a CA, the risk is that any and every incident may lead to the CA's removal, and thus the best path to avoid that is to not have incidents in the first place. Further, a CA with a pattern of incidents is not wrong to be even more careful when it comes to presenting new incidents, especially if they realize that they share similar root causes or further demonstrate problematic patterns. That's not to say that if you only had a single incident, you wouldn't be removed - as the policies capture, any reason and no reason - but on the balance, it has historically tended to be less-likely that first-time incidents lead to removal. When incidents happen, it becomes necessary for Root Programs and the communities they represent or collaborate with to evaluate the details of the incident, as part of making a determination about what next steps are appropriate. This involves investigating what the underlying root causes are, to both ensure that the current CA with the incident understands the significance, the severity, and the steps to remediate it, as well as to help the industry at large develop and learn best practices, to prevent future incidents. We're not the only industry to do this - in many ways, it's borrowed from the aviation industry, that recognizes that critical safety functions deserve thoughtful and detailed analysis to prevent harm coming to those that trust in them. Incident reports also serve to triage the issues - to work and identify the risks and make sure they're being mitigated in a timely fashion. Sometimes, the mitigation of risk may be to remove trust in the CA, other times, there may be less significant steps that can be taken to address both the immediate problem and the underlying issues. DigiCert is now in a precarious position. As a CA, it knows that every one of its Subscribers have agreed, in some legally binding form, that if the CA has misissued a certificate, that it MUST be revoked within 24 hours (or, very recently, and only in some cases, 5 days). The CA has a duty and obligation to their customers, the Subscribers, to make sure that they understa
RE: Underscore characters
I can break down the date by customer. April 30 was the last date for all customers. The actual revocation occurs sometime between Jan 15th and April 30th (still working on a per cert basis to determine this). Note that we actually have the 30 day option available and are recommending it as a remediation instead of an exception. We’d prefer customers to move to the 30 day cert sooner if they can replace the cert but not change the domain name. I’ll file a separate incident report per company. Thanks! Jeremy From: Wayne Thayer Sent: Thursday, December 20, 2018 12:25 PM To: Ryan Sleevi Cc: Jeremy Rowley ; mozilla-dev-security-policy Subject: Re: Underscore characters Jeremy, It's good to hear that you do believe you can provide the necessary level of information prior to 15-Jan. Given that, I'm now thinking of this as if it were a normal incident except that we're moving the reporting prior to the incident actually occurring. With 15 affected customers, and perhaps many more deployment scenarios, I would ask you to break this into separate incident reports per customer as Ryan has previously suggested/requested. I can understand the desire to not have 15 separate compliance bugs filed under DigiCert for what is arguably the same issue, but I think that reporting separately per customer will help to ensure that we receive the level of detail needed to assess the hypothetical incident. I'm also not a fan of the proposed 30-April exceptional revocation deadline. This provides zero opportunity to employ the 30-day cert option if something is missed, and it seems to be an arbitrary date rather than an evaluation of the earliest date by which each customer can safely replace their non-compliant certificates. - Wayne On Thu, Dec 20, 2018 at 11:16 AM Ryan Sleevi via dev-security-policy mailto:dev-security-policy@lists.mozilla.org> > wrote: Thanks for filing this, Jeremy. If I understand correctly, the request DigiCert is asking is: "If we submitted this as an incident report, would it be likely that conversations about distrusting DigiCert would begin?", and that's what you're trying to gauge from the community? I think Wayne's already captured the "We need more information", but I think it may be helpful to explain the reasoning and thinking here. The Baseline Requirements and Root Program policies exist for a purpose: To provide a consistent set of expectations for CAs, which meet the security needs of the products using or operating those policies. As these policies tend to call out, a CA may be removed (distrusted) for any reason or no reason at all - it's entirely at the Program discretion. That said, history tends to largely see removals for patterns of issues that, in aggregate, demonstrate an ongoing and significant risk to users and the Internet at large, although there have been CAs removed for single incidents in the past - such as key compromise or issuing MITM certificates. As a CA, the risk is that any and every incident may lead to the CA's removal, and thus the best path to avoid that is to not have incidents in the first place. Further, a CA with a pattern of incidents is not wrong to be even more careful when it comes to presenting new incidents, especially if they realize that they share similar root causes or further demonstrate problematic patterns. That's not to say that if you only had a single incident, you wouldn't be removed - as the policies capture, any reason and no reason - but on the balance, it has historically tended to be less-likely that first-time incidents lead to removal. When incidents happen, it becomes necessary for Root Programs and the communities they represent or collaborate with to evaluate the details of the incident, as part of making a determination about what next steps are appropriate. This involves investigating what the underlying root causes are, to both ensure that the current CA with the incident understands the significance, the severity, and the steps to remediate it, as well as to help the industry at large develop and learn best practices, to prevent future incidents. We're not the only industry to do this - in many ways, it's borrowed from the aviation industry, that recognizes that critical safety functions deserve thoughtful and detailed analysis to prevent harm coming to those that trust in them. Incident reports also serve to triage the issues - to work and identify the risks and make sure they're being mitigated in a timely fashion. Sometimes, the mitigation of risk may be to remove trust in the CA, other times, there may be less significant steps that can be taken to address both the immediate problem and the underlying issues. DigiCert is now in a precarious position. As a CA, it knows that every one of its Subscribers have agreed, in some legally binding form, that if the CA has misissued a certificate, that it MUST be revoked within 24 hours (or, very recent
Re: Underscore characters
Thanks for filing this, Jeremy. If I understand correctly, the request DigiCert is asking is: "If we submitted this as an incident report, would it be likely that conversations about distrusting DigiCert would begin?", and that's what you're trying to gauge from the community? I think Wayne's already captured the "We need more information", but I think it may be helpful to explain the reasoning and thinking here. The Baseline Requirements and Root Program policies exist for a purpose: To provide a consistent set of expectations for CAs, which meet the security needs of the products using or operating those policies. As these policies tend to call out, a CA may be removed (distrusted) for any reason or no reason at all - it's entirely at the Program discretion. That said, history tends to largely see removals for patterns of issues that, in aggregate, demonstrate an ongoing and significant risk to users and the Internet at large, although there have been CAs removed for single incidents in the past - such as key compromise or issuing MITM certificates. As a CA, the risk is that any and every incident may lead to the CA's removal, and thus the best path to avoid that is to not have incidents in the first place. Further, a CA with a pattern of incidents is not wrong to be even more careful when it comes to presenting new incidents, especially if they realize that they share similar root causes or further demonstrate problematic patterns. That's not to say that if you only had a single incident, you wouldn't be removed - as the policies capture, any reason and no reason - but on the balance, it has historically tended to be less-likely that first-time incidents lead to removal. When incidents happen, it becomes necessary for Root Programs and the communities they represent or collaborate with to evaluate the details of the incident, as part of making a determination about what next steps are appropriate. This involves investigating what the underlying root causes are, to both ensure that the current CA with the incident understands the significance, the severity, and the steps to remediate it, as well as to help the industry at large develop and learn best practices, to prevent future incidents. We're not the only industry to do this - in many ways, it's borrowed from the aviation industry, that recognizes that critical safety functions deserve thoughtful and detailed analysis to prevent harm coming to those that trust in them. Incident reports also serve to triage the issues - to work and identify the risks and make sure they're being mitigated in a timely fashion. Sometimes, the mitigation of risk may be to remove trust in the CA, other times, there may be less significant steps that can be taken to address both the immediate problem and the underlying issues. DigiCert is now in a precarious position. As a CA, it knows that every one of its Subscribers have agreed, in some legally binding form, that if the CA has misissued a certificate, that it MUST be revoked within 24 hours (or, very recently, and only in some cases, 5 days). The CA has a duty and obligation to their customers, the Subscribers, to make sure that they understand this. This is not about a punitive measure or punishing those users for something their CA did - it's because the fundamental and inherent risk is that there are incidents where certificates will need to be replaced in as little as 24 hours, up to and including trust in the CA being removed. To go back to that aviation analogy, the reason planes have maintenance schedules is not because they're going to completely come unglued and fall apart if you miss that maintenance schedule by a day - but because of the severe and significant harm that comes about from having no maintenance schedule at all, or even simply one that just isn't suitable for the risks (to life, property, and safety). Matt Palmer's reply earlier in this thread further expands on some of the other risks here and the hazards that come with. At the same time, DigiCert is, on behalf of their customers, saying that even though both DigiCert and their customers agreed to the 24-hour revocation rule, there are circumstances and situations that make that risky. Despite being an industry standard (as captured in the Baseline Requirements), and despite these agreements, DigiCert is concerned that there are consequences for these customers that did not take adequate precautions to meet the expectations they agreed to, and is trying to perform a risk analysis. Further, they're looking for feedback from the community to make sure that their analysis of the risk - the disruption to their customers - is significant enough that it warrants both the immediate risk of not revoking, the business risk to DigiCert, and the lasting risk to the ecosystem, in intentionally violating the BRs. It's not my intent to sound harsh, but to make sure it's clearly and unambiguously stated as to what's happening. The reason for doing this
RE: Underscore characters
Thanks Wayne. Happy to update with that information. We’ll try to provide it all be end of the year, definitely before Jan 12. I can answer two of these generally now: * Reason that publicly-trusted certificates are in use - They are used on websites and infrastructure accessed through browsers. There are some that don’t need to be publicly trusted certificates, but the systems still check revocation. If Mozilla blocked the certs, the impact is minimal. If the certificates are revoke, the infrastructure goes down. Would you like me to identify which ones could be blocked by Mozilla vs. revoked? * Reason that the provision for 30-day certificates isn't helpful - The blackout periods don’t allow any change in infrastructure so replacing the certificates is just as difficult as changing the domain names. Even large non-tech companies don’t have a ton of automation so it’s not surprising that they’ll need more days to replace the certs if their blackout period ends the 12th. Of course, I’ll provide further information in the incident report as more specific information comes in. From: Wayne Thayer Sent: Thursday, December 20, 2018 9:04 AM To: Jeremy Rowley Cc: r...@sleevi.com; mozilla-dev-security-policy Subject: Re: Underscore characters Jeremy, On Wed, Dec 19, 2018 at 10:55 PM Jeremy Rowley via dev-security-policy mailto:dev-security-policy@lists.mozilla.org> > wrote: Done: https://bugzilla.mozilla.org/show_bug.cgi?id=1515564 Thanks for submitting this. It ended up being about 1200 certs total that we are hearing can’t be replaced because of blackout periods. These 1200 are only the ones that can't be replaced by Jan 15th and will cause outages if revoked then? I don't think the information you've supplied is anywhere close to what Ryan asked for or what the community needs in order to make the decision you're asking for. I'm looking for specifics on why every cohort (i.e. every deployment scenario for every customer requesting an extension) of these certificates can't be revoked, such as: * Specific per-customer change freeze dates and the rationale for them * Explanations of the effort and risk involved in replacing them * Reason that publicly-trusted certificates are in use * Reason that the provision for 30-day certificates isn't helpful Only with this information can we have some assurance that any exceptions are limited to the bare minimum and that we're able to learn and improve. Without this information, we're still in the situation of blindly trusting DigiCert to do the right thing, which is no different than having a CA report an incident after the fact. Is it realistic to expect that you can provide the level of detail that Ryan and I are requesting prior to 15-Jan? From: Ryan Sleevi mailto:r...@sleevi.com> > Sent: Wednesday, December 19, 2018 11:05 AM To: Jeremy Rowley mailto:jeremy.row...@digicert.com> > Cc: r...@sleevi.com <mailto:r...@sleevi.com> ; mozilla-dev-security-policy mailto:mozilla-dev-security-pol...@lists.mozilla.org> > Subject: Re: Underscore characters Look forward to seeing and discussing once the full scope of the request is shared. On Wed, Dec 19, 2018 at 12:21 PM Jeremy Rowley mailto:jeremy.row...@digicert.com> <mailto:jeremy.row...@digicert.com <mailto:jeremy.row...@digicert.com> > > wrote: We will post the full list of exceptions today. One of the big factors should be the risk to the industry/community if the certificates aren’t revoked. Perhaps we can identify what the risk to the community is in revocation delays first? There’s no need to know the exact certs to talk about what the risk associated with underscore characters is. Could you please explain the risk to the community in a revocation delay as the “unreasonable” argument isn’t really supported without that understanding. smime.p7s Description: S/MIME cryptographic signature ___ dev-security-policy mailing list dev-security-policy@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-security-policy
Re: Underscore characters
Jeremy, On Wed, Dec 19, 2018 at 10:55 PM Jeremy Rowley via dev-security-policy < dev-security-policy@lists.mozilla.org> wrote: > Done: > > > > https://bugzilla.mozilla.org/show_bug.cgi?id=1515564 > > Thanks for submitting this. > > > It ended up being about 1200 certs total that we are hearing can’t be > replaced because of blackout periods. > > These 1200 are only the ones that can't be replaced by Jan 15th and will cause outages if revoked then? I don't think the information you've supplied is anywhere close to what Ryan asked for or what the community needs in order to make the decision you're asking for. I'm looking for specifics on why every cohort (i.e. every deployment scenario for every customer requesting an extension) of these certificates can't be revoked, such as: * Specific per-customer change freeze dates and the rationale for them * Explanations of the effort and risk involved in replacing them * Reason that publicly-trusted certificates are in use * Reason that the provision for 30-day certificates isn't helpful Only with this information can we have some assurance that any exceptions are limited to the bare minimum and that we're able to learn and improve. Without this information, we're still in the situation of blindly trusting DigiCert to do the right thing, which is no different than having a CA report an incident after the fact. Is it realistic to expect that you can provide the level of detail that Ryan and I are requesting prior to 15-Jan? > From: Ryan Sleevi > Sent: Wednesday, December 19, 2018 11:05 AM > To: Jeremy Rowley > Cc: r...@sleevi.com; mozilla-dev-security-policy < > mozilla-dev-security-pol...@lists.mozilla.org> > Subject: Re: Underscore characters > > > > Look forward to seeing and discussing once the full scope of the request > is shared. > > > > On Wed, Dec 19, 2018 at 12:21 PM Jeremy Rowley <mailto:jeremy.row...@digicert.com> > wrote: > > We will post the full list of exceptions today. > > > > One of the big factors should be the risk to the industry/community if the > certificates aren’t revoked. Perhaps we can identify what the risk to the > community is in revocation delays first? There’s no need to know the exact > certs to talk about what the risk associated with underscore characters is. > Could you please explain the risk to the community in a revocation delay as > the “unreasonable” argument isn’t really supported without that > understanding. > > ___ dev-security-policy mailing list dev-security-policy@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-security-policy
RE: Underscore characters
Done: https://bugzilla.mozilla.org/show_bug.cgi?id=1515564 It ended up being about 1200 certs total that we are hearing can’t be replaced because of blackout periods. From: Ryan Sleevi Sent: Wednesday, December 19, 2018 11:05 AM To: Jeremy Rowley Cc: r...@sleevi.com; mozilla-dev-security-policy Subject: Re: Underscore characters Look forward to seeing and discussing once the full scope of the request is shared. On Wed, Dec 19, 2018 at 12:21 PM Jeremy Rowley mailto:jeremy.row...@digicert.com> > wrote: We will post the full list of exceptions today. One of the big factors should be the risk to the industry/community if the certificates aren’t revoked. Perhaps we can identify what the risk to the community is in revocation delays first? There’s no need to know the exact certs to talk about what the risk associated with underscore characters is. Could you please explain the risk to the community in a revocation delay as the “unreasonable” argument isn’t really supported without that understanding. From: Ryan Sleevi mailto:r...@sleevi.com> > Sent: Wednesday, December 19, 2018 7:17 AM To: Jeremy Rowley mailto:jeremy.row...@digicert.com> > Cc: r...@sleevi.com <mailto:r...@sleevi.com> ; mozilla-dev-security-policy mailto:mozilla-dev-security-pol...@lists.mozilla.org> > Subject: Re: Underscore characters While I appreciate you sharing what you have, as I tried to capture in my previous message, I don't believe there can be any discussion or consideration in earnest without the full and final information. I don't think it's reasonable to drip in information piece meal, given the impact and affect that can have - whether incomplete information for the issue or whether additional customers being added. You're making a huge request of the community, arguably one that borderlines unreasonable given the set of issues had in the past. I do want to help you achieve your goal of understanding what that would look like, but that's only possible with full and complete information. You mentioned it's roughly 15 companies. If you had ten committed, but were waiting on the remaining five to give the OK, I think it would be irresponsible to hold up having that conversation until you get the OK. Quite simply, if you don't get the OK from those five companies, then we shouldn't even be discussing them. Ultimately, the ball is in your court as to how you want to address this with your customers, but I think that delaying the conversation in order to make sure "stragglers" are included is probably not the wisest for your customers that have their stuff together. As such, I don't think the conversation can begin without that (hypothetical) incident report, and I look forward to you deciding what that scope will be in order to share and commit to it. On Tue, Dec 18, 2018 at 9:51 PM Jeremy Rowley mailto:jeremy.row...@digicert.com> > wrote: Yeah – I’ll be providing an accurate incident report (working on gathering all the information). The incident report assumes we don’t revoke of course. Revocation is still on the table. However, I wanted to start the conversation with everything I know so far: 1) ~2200 certs 2) Roughly 15 companies 3) Only one has publicly chimed in so far on the Mozilla thread (still hoping more will…) 4) Revocation of all certs will occur by May 1, 2019, depending on how the discussion here goes. 5) The common thread is that the Jan 15th deadline falls in the blackout window of most orgs. They generally come off it between Jan 15-Feb 15. They can’t replace the cert or change the domain so the 30 day cert option doesn’t help. 6) We provided notice as soon as the ballot passed. We blocked issuance prior to that date, but we’d hoped that the certs could remain valid until expiration. We had trouble with our BI providing the information so some notices went out later than I’d hoped. I’ll find the exact date on when all notices were complete. Ballot 202 failed. I’m not sure how it’s relevant other than to indicate there was definite disagreement about whether underscores were permitted or not. As previously mentioned, I didn’t consider underscore characters prohibited until the ballot was proposed eliminating them in Oct. I know the general Mozilla population disagrees but, right or wrong, that’s the root cause of it all. I can explain my reasoning again here, but I doubt it materially alters the conversation and outcome. From: Ryan Sleevi mailto:r...@sleevi.com> > Sent: Tuesday, December 18, 2018 7:35 PM To: Jeremy Rowley mailto:jeremy.row...@digicert.com> > Cc: mozilla-dev-security-policy mailto:mozilla-dev-security-pol...@lists.mozilla.org> > Subject: Re: Underscore characters Jeremy, It seems like any answer for what it "might" look like if a CA violated the BRs in a particular way is going to be predicated on what th
Re: Underscore characters
On Wed, Dec 19, 2018 at 05:20:59PM +, Jeremy Rowley via dev-security-policy wrote: > One of the big factors should be the risk to the industry/community if the > certificates aren’t revoked. Perhaps we can identify what the risk to the > community is in revocation delays first? There’s no need to know the > exact certs to talk about what the risk associated with underscore > characters is. Could you please explain the risk to the community in a > revocation delay as the “unreasonable” argument isn’t really supported > without that understanding. I think an important risk to the community of not revoking as per the CA/B Forum's accepted ballot timeline is sending the message that the rules of the game are optional, and there is commercial benefit to be gained in not following the rules. I'm sure there are some CAs whose systems were always setup in a standards-compliant fashion, and refused to issue certificates for invalid names. I'm equally sure that at least one of those CAs lost a sale over the years as a result of that, and that sale went to a competitor which was *not* adhering to the standards. Now the issue has been raised, clarification made, and a decision has been made as to how to move forward. To provide further benefit (in the form of a waiver from the agreed-upon rules) to CAs which have failed to follow the rules in the past does not encourage adherence in the future. Certainly, if I were the CEO of a for-profit CA which *had* followed the no-underscores rule, I might be inclined to gently encourage my developers to play a little faster and looser with their interpretations in the future, if it would provide my organisation with a revenue benefit, and it was clear that there was to be no meaningful negative consequence *to me* as a result. To do otherwise would actually be contrary to the stated goals of the organisation. Please don't misunderstand my words to think that I'm saying that any CA *deliberately* ignored the standards around underscores in order to sell some more certs. I'm well aware that the rules around valid hostnames, domain names, DNS labels, etc are not the clearest, and most people wouldn't read them even if they were. It's undeniable, though, that CAs which allowed underscores in places that are supposed to be valid LDH domain names made a mistake, and to deliberately misquote Jurassic Park's John Hammond, "I don't blame people for their mistakes, but I do expect them to take responsibility for them." To not expect CAs to take responsibility for their mistakes sends a *terrible* message to the entire ecosystem, one that would have far greater long-term repurcussions than any isolated harm from the presence of underscores themselves. Whilst it's not quite the textbook definition, part of the Wikipedia page on "Moral Hazard" says, "when a person takes more risks because someone else bears the cost of those risks". That's a pretty reasonable expression of what's going on here. - Matt ___ dev-security-policy mailing list dev-security-policy@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-security-policy
RE: Underscore characters
We will post the full list of exceptions today. One of the big factors should be the risk to the industry/community if the certificates aren’t revoked. Perhaps we can identify what the risk to the community is in revocation delays first? There’s no need to know the exact certs to talk about what the risk associated with underscore characters is. Could you please explain the risk to the community in a revocation delay as the “unreasonable” argument isn’t really supported without that understanding. From: Ryan Sleevi Sent: Wednesday, December 19, 2018 7:17 AM To: Jeremy Rowley Cc: r...@sleevi.com; mozilla-dev-security-policy Subject: Re: Underscore characters While I appreciate you sharing what you have, as I tried to capture in my previous message, I don't believe there can be any discussion or consideration in earnest without the full and final information. I don't think it's reasonable to drip in information piece meal, given the impact and affect that can have - whether incomplete information for the issue or whether additional customers being added. You're making a huge request of the community, arguably one that borderlines unreasonable given the set of issues had in the past. I do want to help you achieve your goal of understanding what that would look like, but that's only possible with full and complete information. You mentioned it's roughly 15 companies. If you had ten committed, but were waiting on the remaining five to give the OK, I think it would be irresponsible to hold up having that conversation until you get the OK. Quite simply, if you don't get the OK from those five companies, then we shouldn't even be discussing them. Ultimately, the ball is in your court as to how you want to address this with your customers, but I think that delaying the conversation in order to make sure "stragglers" are included is probably not the wisest for your customers that have their stuff together. As such, I don't think the conversation can begin without that (hypothetical) incident report, and I look forward to you deciding what that scope will be in order to share and commit to it. On Tue, Dec 18, 2018 at 9:51 PM Jeremy Rowley mailto:jeremy.row...@digicert.com> > wrote: Yeah – I’ll be providing an accurate incident report (working on gathering all the information). The incident report assumes we don’t revoke of course. Revocation is still on the table. However, I wanted to start the conversation with everything I know so far: 1) ~2200 certs 2) Roughly 15 companies 3) Only one has publicly chimed in so far on the Mozilla thread (still hoping more will…) 4) Revocation of all certs will occur by May 1, 2019, depending on how the discussion here goes. 5) The common thread is that the Jan 15th deadline falls in the blackout window of most orgs. They generally come off it between Jan 15-Feb 15. They can’t replace the cert or change the domain so the 30 day cert option doesn’t help. 6) We provided notice as soon as the ballot passed. We blocked issuance prior to that date, but we’d hoped that the certs could remain valid until expiration. We had trouble with our BI providing the information so some notices went out later than I’d hoped. I’ll find the exact date on when all notices were complete. Ballot 202 failed. I’m not sure how it’s relevant other than to indicate there was definite disagreement about whether underscores were permitted or not. As previously mentioned, I didn’t consider underscore characters prohibited until the ballot was proposed eliminating them in Oct. I know the general Mozilla population disagrees but, right or wrong, that’s the root cause of it all. I can explain my reasoning again here, but I doubt it materially alters the conversation and outcome. From: Ryan Sleevi mailto:r...@sleevi.com> > Sent: Tuesday, December 18, 2018 7:35 PM To: Jeremy Rowley mailto:jeremy.row...@digicert.com> > Cc: mozilla-dev-security-policy mailto:mozilla-dev-security-pol...@lists.mozilla.org> > Subject: Re: Underscore characters Jeremy, It seems like any answer for what it "might" look like if a CA violated the BRs in a particular way is going to be predicated on what the incident report says. In the case of a hypothetical like this, it seems like the hypothetical incident report would discuss what is planned or proposed, and should a CA go forward with such an intentional violation, the 'actual' incident report would equally consider how accurate that was. Recall that the approach to incident reporting is not punitive - it's to make sure that we're addressing systemic gaps and issues, that we've understood the issues, and have the available data to assess the impact, risk, and any patterns of issues. The incident reporting template is one way to provide that data in a structured way and to gather feedback. I think a minimum next step is to move f
Re: Underscore characters
While I appreciate you sharing what you have, as I tried to capture in my previous message, I don't believe there can be any discussion or consideration in earnest without the full and final information. I don't think it's reasonable to drip in information piece meal, given the impact and affect that can have - whether incomplete information for the issue or whether additional customers being added. You're making a huge request of the community, arguably one that borderlines unreasonable given the set of issues had in the past. I do want to help you achieve your goal of understanding what that would look like, but that's only possible with full and complete information. You mentioned it's roughly 15 companies. If you had ten committed, but were waiting on the remaining five to give the OK, I think it would be irresponsible to hold up having that conversation until you get the OK. Quite simply, if you don't get the OK from those five companies, then we shouldn't even be discussing them. Ultimately, the ball is in your court as to how you want to address this with your customers, but I think that delaying the conversation in order to make sure "stragglers" are included is probably not the wisest for your customers that have their stuff together. As such, I don't think the conversation can begin without that (hypothetical) incident report, and I look forward to you deciding what that scope will be in order to share and commit to it. On Tue, Dec 18, 2018 at 9:51 PM Jeremy Rowley wrote: > Yeah – I’ll be providing an accurate incident report (working on gathering > all the information). The incident report assumes we don’t revoke of > course. Revocation is still on the table. However, I wanted to start the > conversation with everything I know so far: > > 1) ~2200 certs > > 2) Roughly 15 companies > 3) Only one has publicly chimed in so far on the Mozilla thread (still > hoping more will…) > > 4) Revocation of all certs will occur by May 1, 2019, depending on how the > discussion here goes. > 5) The common thread is that the Jan 15th deadline falls in the blackout > window of most orgs. They generally come off it between Jan 15-Feb 15. They > can’t replace the cert or change the domain so the 30 day cert option > doesn’t help. > > 6) We provided notice as soon as the ballot passed. We blocked issuance > prior to that date, but we’d hoped that the certs could remain valid until > expiration. We had trouble with our BI providing the information so some > notices went out later than I’d hoped. I’ll find the exact date on when all > notices were complete. > > > > Ballot 202 failed. I’m not sure how it’s relevant other than to indicate > there was definite disagreement about whether underscores were permitted or > not. As previously mentioned, I didn’t consider underscore characters > prohibited until the ballot was proposed eliminating them in Oct. I know > the general Mozilla population disagrees but, right or wrong, that’s the > root cause of it all. I can explain my reasoning again here, but I doubt it > materially alters the conversation and outcome. > > > > *From:* Ryan Sleevi > *Sent:* Tuesday, December 18, 2018 7:35 PM > *To:* Jeremy Rowley > *Cc:* mozilla-dev-security-policy < > mozilla-dev-security-pol...@lists.mozilla.org> > *Subject:* Re: Underscore characters > > > > Jeremy, > > > > It seems like any answer for what it "might" look like if a CA violated > the BRs in a particular way is going to be predicated on what the incident > report says. In the case of a hypothetical like this, it seems like the > hypothetical incident report would discuss what is planned or proposed, and > should a CA go forward with such an intentional violation, the 'actual' > incident report would equally consider how accurate that was. > > > > Recall that the approach to incident reporting is not punitive - it's to > make sure that we're addressing systemic gaps and issues, that we've > understood the issues, and have the available data to assess the impact, > risk, and any patterns of issues. The incident reporting template is one > way to provide that data in a structured way and to gather feedback. > > > > I think a minimum next step is to move from the abstract discussion to the > concrete: imagine you went forward on Jan 15 and had to file an incident > report. Write the report like that. Include the timeframes, affected > certificates, impact, root causes, remediation plans, etc. Having a > complete presentation of what the discussion is about seems critical to > having that discussion, because it would be unreasonable to expect > information to trickle in and new customers or use cases added as the > discussion progresses. > > > > Thus there's a balance
Re: Underscore characters
On 19/12/2018 04:14, Peter Bowen wrote: > On Tue, Dec 18, 2018 at 6:52 PM Jeremy Rowley via dev-security-policy < > dev-security-policy@lists.mozilla.org> wrote: > >> Ballot 202 failed. I’m not sure how it’s relevant other than to indicate >> there was definite disagreement about whether underscores were permitted or >> not. As previously mentioned, I didn’t consider underscore characters >> prohibited until the ballot was proposed eliminating them in Oct. I know >> the general Mozilla population disagrees but, right or wrong, that’s the >> root cause of it all. I can explain my reasoning again here, but I doubt it >> materially alters the conversation and outcome. >> > > I agree that Jeremy that the situation with underscores was unclear prior > to the ballot in October. Three years ago when I was writing certlint, my > very first public commit has the comment: > # Allow RFC defying '*' and '_' > > I honestly haven't been pay a lot of attention to the CA/Browser Forum > recently. Given the rationale for getting rid of underscores is RFC > compliance, did the ballot also disallow asterisks? They are also not > allowed by the "preferred name syntax", as specified by Section 3.5 of > [RFC1034] <https://tools.ietf.org/html/rfc1034#section-3.5> and as modified > by Section 2.1 of <https://tools.ietf.org/html/rfc1123#section-2.1> > [RFC1123] <https://tools.ietf.org/html/rfc1123#section-2.1>. > The problematic section of RFC5280 contains this paragraph, wedged between encoding descriptions (which happen to include a reference to the "preferred syntax" of host names) and the corresponding ASN.1 syntax: Finally, the semantics of subject alternative names that include wildcard characters (e.g., as a placeholder for a set of names) are not addressed by this specification. Applications with specific requirements MAY use such names, but they must define the semantics. A different RFC defines the modern semantics of wildcard certificates, thus providing the required definition. Enjoy Jakob -- Jakob Bohm, CIO, Partner, WiseMo A/S. https://www.wisemo.com Transformervej 29, 2860 Søborg, Denmark. Direct +45 31 13 16 10 This public discussion message is non-binding and may contain errors. WiseMo - Remote Service Management for PCs, Phones and Embedded ___ dev-security-policy mailing list dev-security-policy@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-security-policy
Re: Underscore characters
On Tue, Dec 18, 2018 at 6:52 PM Jeremy Rowley via dev-security-policy < dev-security-policy@lists.mozilla.org> wrote: > Ballot 202 failed. I’m not sure how it’s relevant other than to indicate > there was definite disagreement about whether underscores were permitted or > not. As previously mentioned, I didn’t consider underscore characters > prohibited until the ballot was proposed eliminating them in Oct. I know > the general Mozilla population disagrees but, right or wrong, that’s the > root cause of it all. I can explain my reasoning again here, but I doubt it > materially alters the conversation and outcome. > I agree that Jeremy that the situation with underscores was unclear prior to the ballot in October. Three years ago when I was writing certlint, my very first public commit has the comment: # Allow RFC defying '*' and '_' I honestly haven't been pay a lot of attention to the CA/Browser Forum recently. Given the rationale for getting rid of underscores is RFC compliance, did the ballot also disallow asterisks? They are also not allowed by the "preferred name syntax", as specified by Section 3.5 of [RFC1034] <https://tools.ietf.org/html/rfc1034#section-3.5> and as modified by Section 2.1 of <https://tools.ietf.org/html/rfc1123#section-2.1> [RFC1123] <https://tools.ietf.org/html/rfc1123#section-2.1>. Thanks, Peter ___ dev-security-policy mailing list dev-security-policy@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-security-policy
RE: Underscore characters
Yeah – I’ll be providing an accurate incident report (working on gathering all the information). The incident report assumes we don’t revoke of course. Revocation is still on the table. However, I wanted to start the conversation with everything I know so far: 1) ~2200 certs 2) Roughly 15 companies 3) Only one has publicly chimed in so far on the Mozilla thread (still hoping more will…) 4) Revocation of all certs will occur by May 1, 2019, depending on how the discussion here goes. 5) The common thread is that the Jan 15th deadline falls in the blackout window of most orgs. They generally come off it between Jan 15-Feb 15. They can’t replace the cert or change the domain so the 30 day cert option doesn’t help. 6) We provided notice as soon as the ballot passed. We blocked issuance prior to that date, but we’d hoped that the certs could remain valid until expiration. We had trouble with our BI providing the information so some notices went out later than I’d hoped. I’ll find the exact date on when all notices were complete. Ballot 202 failed. I’m not sure how it’s relevant other than to indicate there was definite disagreement about whether underscores were permitted or not. As previously mentioned, I didn’t consider underscore characters prohibited until the ballot was proposed eliminating them in Oct. I know the general Mozilla population disagrees but, right or wrong, that’s the root cause of it all. I can explain my reasoning again here, but I doubt it materially alters the conversation and outcome. From: Ryan Sleevi Sent: Tuesday, December 18, 2018 7:35 PM To: Jeremy Rowley Cc: mozilla-dev-security-policy Subject: Re: Underscore characters Jeremy, It seems like any answer for what it "might" look like if a CA violated the BRs in a particular way is going to be predicated on what the incident report says. In the case of a hypothetical like this, it seems like the hypothetical incident report would discuss what is planned or proposed, and should a CA go forward with such an intentional violation, the 'actual' incident report would equally consider how accurate that was. Recall that the approach to incident reporting is not punitive - it's to make sure that we're addressing systemic gaps and issues, that we've understood the issues, and have the available data to assess the impact, risk, and any patterns of issues. The incident reporting template is one way to provide that data in a structured way and to gather feedback. I think a minimum next step is to move from the abstract discussion to the concrete: imagine you went forward on Jan 15 and had to file an incident report. Write the report like that. Include the timeframes, affected certificates, impact, root causes, remediation plans, etc. Having a complete presentation of what the discussion is about seems critical to having that discussion, because it would be unreasonable to expect information to trickle in and new customers or use cases added as the discussion progresses. Thus there's a balance to be struck: Treating each hypothetical as a "separate" incident report runs the risk of being considered in isolation, ignoring both the systemic gaps and the cumulative risk. At the same time, treating it as a "singular" incident report tries to paint all problems in the same stroke, and can overlook distinct systemic issues. Both cases run the risk of "scope creep", which is constantly adding or expanding the scope, which is as well received in legitimate incident reports as it is hypothetical (which is to say: not well). Perhaps the best analogy is to that of subordinate CAs: each time a subordinate has an issue, that's an incident report, and a pattern of issues at distinct subordinates is equally a concerning issue for the parent CA. You don't want to loop all distinct subordinates into one issue, but you also don't want to lose sight of the systemic issues with the parent. Beyond that framing and execution, it seems useful to suggest that any timeline about underscores should at least acknowledge Ballot 202 in June 2017 and any/all steps the CA took leading up to and following SC12. None of this is radically new or should be surprising: DigiCert and other CAs have already had similar conversations in discussing other matters of BR compliance and revocation. All of these have become part of the CA's record of incidents. When the CA proposes extending revocation timelines, a discussion of the facts, risks, scope, and patterns play a core part in any discussion in determining the short term acceptability of the proposal, and unquestionably all factor in to any long-term discussions that may later happen. The one last closing thought is that I think we're in the waning days for when such hypothetical issues or concrete delay proposals can or should be discussed. Given the many discussions that have be
Re: Underscore characters
Jeremy, It seems like any answer for what it "might" look like if a CA violated the BRs in a particular way is going to be predicated on what the incident report says. In the case of a hypothetical like this, it seems like the hypothetical incident report would discuss what is planned or proposed, and should a CA go forward with such an intentional violation, the 'actual' incident report would equally consider how accurate that was. Recall that the approach to incident reporting is not punitive - it's to make sure that we're addressing systemic gaps and issues, that we've understood the issues, and have the available data to assess the impact, risk, and any patterns of issues. The incident reporting template is one way to provide that data in a structured way and to gather feedback. I think a minimum next step is to move from the abstract discussion to the concrete: imagine you went forward on Jan 15 and had to file an incident report. Write the report like that. Include the timeframes, affected certificates, impact, root causes, remediation plans, etc. Having a complete presentation of what the discussion is about seems critical to having that discussion, because it would be unreasonable to expect information to trickle in and new customers or use cases added as the discussion progresses. Thus there's a balance to be struck: Treating each hypothetical as a "separate" incident report runs the risk of being considered in isolation, ignoring both the systemic gaps and the cumulative risk. At the same time, treating it as a "singular" incident report tries to paint all problems in the same stroke, and can overlook distinct systemic issues. Both cases run the risk of "scope creep", which is constantly adding or expanding the scope, which is as well received in legitimate incident reports as it is hypothetical (which is to say: not well). Perhaps the best analogy is to that of subordinate CAs: each time a subordinate has an issue, that's an incident report, and a pattern of issues at distinct subordinates is equally a concerning issue for the parent CA. You don't want to loop all distinct subordinates into one issue, but you also don't want to lose sight of the systemic issues with the parent. Beyond that framing and execution, it seems useful to suggest that any timeline about underscores should at least acknowledge Ballot 202 in June 2017 and any/all steps the CA took leading up to and following SC12. None of this is radically new or should be surprising: DigiCert and other CAs have already had similar conversations in discussing other matters of BR compliance and revocation. All of these have become part of the CA's record of incidents. When the CA proposes extending revocation timelines, a discussion of the facts, risks, scope, and patterns play a core part in any discussion in determining the short term acceptability of the proposal, and unquestionably all factor in to any long-term discussions that may later happen. The one last closing thought is that I think we're in the waning days for when such hypothetical issues or concrete delay proposals can or should be discussed. Given the many discussions that have been had regarding revocation - regarding technical non-compliance, compromised keys, weak validation, etc - the argument that "replacing a cert is hard" is not really going to be acceptable anymore without demonstration about what steps are being taken by CAs and Subscribers to mitigate that risk (such as automation) and communicate expectations (such as in Subscriber Agreements or Terms of Sale). I don't think we want to go through 2019, and certainly not come out of it, having the same conversations we've been having in 2018. The best way to prevent that is for CAs to take clear steps to work to resolve these issues with their customers, so that it never becomes an issue for them, or their CA, in the first place. CAs that aren't able to demonstrate steps towards that in future discussions are unlikely to be looked upon too favorably if there are future incident reports. On Tue, Dec 18, 2018 at 5:43 PM Jeremy Rowley via dev-security-policy < dev-security-policy@lists.mozilla.org> wrote: > The total number of certs impacted is about 2200. Just more info. > > -Original Message- > From: dev-security-policy > On > Behalf Of Jeremy Rowley via dev-security-policy > Sent: Tuesday, December 18, 2018 3:28 PM > To: mozilla-dev-security-policy > > Subject: Underscore characters > > We're looking at the feasibility of replacing the certificates with > underscore characters by Jan 15th. Revoking all of the certificates will > cause pretty bad outages. We're prepared to revoke them but would like to > discuss (before the date) what should happen if we don't revoke. There are > about 15 customers (which I can't disclose the names yet but am working on > the list of certs)
RE: Underscore characters
The total number of certs impacted is about 2200. Just more info. -Original Message- From: dev-security-policy On Behalf Of Jeremy Rowley via dev-security-policy Sent: Tuesday, December 18, 2018 3:28 PM To: mozilla-dev-security-policy Subject: Underscore characters We're looking at the feasibility of replacing the certificates with underscore characters by Jan 15th. Revoking all of the certificates will cause pretty bad outages. We're prepared to revoke them but would like to discuss (before the date) what should happen if we don't revoke. There are about 15 customers (which I can't disclose the names yet but am working on the list of certs). The number of certificates range between 1700-100 certificates per customer. The primary reason for this is every one of these organization are in a holiday blackout. The blackout periods end between Jan 12-Feb 15. Can we start this discussion now on what this means? I'll provide certificate lists as I have a timeline on when they plan on replacing. Jeremy smime.p7s Description: S/MIME cryptographic signature ___ dev-security-policy mailing list dev-security-policy@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-security-policy
Re: Underscore characters and DigiCert
I also don't think removing these roots is a good solution because: * To Ryan's point, it's too late to actually get them removed on Mozilla's current release cycle. * Again echoing Ryan's comments, there are a lot of consumers of NSS and abruptly yanking these roots to work around the ballot SC12 requirement could be very disruptive, even if it doesn't affect Firefox. We might not approve the request, or at least not quickly enough to meet the goal. * As I understand it, this workaround wouldn't help all DigiCert customers who are having trouble meeting the SC12 deadline. While the numbers would be smaller, DigiCert would still be dealing with an incident (I disagree with Ryan's comments in another thread that each customer for whom revocation is delayed constitutes a separate incident). - Wayne On Thu, Dec 13, 2018 at 4:16 PM Ryan Sleevi wrote: > To expand: You would like to request removal and then it be treated as if > you’re already removed, correct? > > I think for the reasons Wayne outlined, that would be detrimental to the > ecosystem as a whole. For example, the request for removal might actually > be denied (!!!) because of the outsize impact it could have on Mozilla > Firefox/NSS users. > > 1) Mozilla blocks the roots via OneCRL, disrupting all existing > whitelisted certificates. I’m sure Apple would be very displeased by this. > 2) Mozilla removes the roots in a future update. This takes time to > develop, release, and distribute - and during that time, clients with the > ongoing trust (including ESR versions) would be impacted. It also suffers > the Apple problem (for impact), which would need to be addressed. > 3) Mozilla acknowledges the request but doesn’t take steps to remove - > this has all the current downsides, but without any expectations of > DigiCert abiding by the program requirements. > > For that reason, the best answer would seem to be that if DigiCert were to > make that request, Mozilla would deny it, so that DigiCert would be clear > as to the continued expectation to abide by the program requirements, until > such a time as a workable solution was identified, reviewed, developed and > implemented, and shipped. > > Of course, that no doubt sounds a bit odd, so if denying wasn’t an option > - that is, if it had to be treated like a compromised root with an > emergency update of some form - then it seems the answer should also be to > treat the organization as one that can’t control their root, and treat it > like an emergency update. > > These are all example responses to your specific request, and a light > exploration about some of the consequences. I have no doubt that, as a CA, > it’d be preferable to know exactly what would or could happen in these root > removal requests. I don’t believe the SHA-1 process for those was at all > beneficial for the community, and it had seemed like CAs had recognized > that and would be unlikely to do it again. If we think this is something to > formalize, then approaches like Microsoft’s - which contractually forbid > it, add timelines for processing, and set expectations for ongoing audits > for years after the “removal” in order to protect users, is a better model. > In the context of Mozilla, which lacks such a contractual lever in its > policy, I would suggest an approach that treats such requests as: > 1) Best effort to remove individual roots, with an expectation that audits > will continue until it is “safe” to stop and all policy obligations met > 2) If the above is breached/violated, then treat it as removing all of > that organizations and affiliates roots (that is, treating it like > organization compromise) > > Hopefully, given the reasons above, that seems like an eminently > reasonable policy position to take. Deliberately introducing issues like > that merely externalizes the risks, and so there needs to be appropriate > balances - whether contractual, financial, or political - to offset those > externalized costs. Similarly, allowing such actions makes the inclusion of > a root that much more dangerous, and thus gives further incentive to deny > organizations that have demonstrated such patterns future roots, given the > risks. > > On Thu, Dec 13, 2018 at 5:15 PM Jeremy Rowley via dev-security-policy < > dev-security-policy@lists.mozilla.org> wrote: > >> Can we request removal of these roots now? This seems very similar to the >> SHA1 situation where CAs requested root removal and then treated the root >> as >> private, regardless of the trust in older platforms. >> >> -Original Message- >> From: dev-security-policy >> On >> Behalf Of Wayne Thayer via dev-security-policy >> Sent: Thursday, December 13, 2018 3:11 PM >> To: mozilla-dev-security-policy >
Re: Underscore characters and DigiCert
To expand: You would like to request removal and then it be treated as if you’re already removed, correct? I think for the reasons Wayne outlined, that would be detrimental to the ecosystem as a whole. For example, the request for removal might actually be denied (!!!) because of the outsize impact it could have on Mozilla Firefox/NSS users. 1) Mozilla blocks the roots via OneCRL, disrupting all existing whitelisted certificates. I’m sure Apple would be very displeased by this. 2) Mozilla removes the roots in a future update. This takes time to develop, release, and distribute - and during that time, clients with the ongoing trust (including ESR versions) would be impacted. It also suffers the Apple problem (for impact), which would need to be addressed. 3) Mozilla acknowledges the request but doesn’t take steps to remove - this has all the current downsides, but without any expectations of DigiCert abiding by the program requirements. For that reason, the best answer would seem to be that if DigiCert were to make that request, Mozilla would deny it, so that DigiCert would be clear as to the continued expectation to abide by the program requirements, until such a time as a workable solution was identified, reviewed, developed and implemented, and shipped. Of course, that no doubt sounds a bit odd, so if denying wasn’t an option - that is, if it had to be treated like a compromised root with an emergency update of some form - then it seems the answer should also be to treat the organization as one that can’t control their root, and treat it like an emergency update. These are all example responses to your specific request, and a light exploration about some of the consequences. I have no doubt that, as a CA, it’d be preferable to know exactly what would or could happen in these root removal requests. I don’t believe the SHA-1 process for those was at all beneficial for the community, and it had seemed like CAs had recognized that and would be unlikely to do it again. If we think this is something to formalize, then approaches like Microsoft’s - which contractually forbid it, add timelines for processing, and set expectations for ongoing audits for years after the “removal” in order to protect users, is a better model. In the context of Mozilla, which lacks such a contractual lever in its policy, I would suggest an approach that treats such requests as: 1) Best effort to remove individual roots, with an expectation that audits will continue until it is “safe” to stop and all policy obligations met 2) If the above is breached/violated, then treat it as removing all of that organizations and affiliates roots (that is, treating it like organization compromise) Hopefully, given the reasons above, that seems like an eminently reasonable policy position to take. Deliberately introducing issues like that merely externalizes the risks, and so there needs to be appropriate balances - whether contractual, financial, or political - to offset those externalized costs. Similarly, allowing such actions makes the inclusion of a root that much more dangerous, and thus gives further incentive to deny organizations that have demonstrated such patterns future roots, given the risks. On Thu, Dec 13, 2018 at 5:15 PM Jeremy Rowley via dev-security-policy < dev-security-policy@lists.mozilla.org> wrote: > Can we request removal of these roots now? This seems very similar to the > SHA1 situation where CAs requested root removal and then treated the root > as > private, regardless of the trust in older platforms. > > -Original Message- > From: dev-security-policy > On > Behalf Of Wayne Thayer via dev-security-policy > Sent: Thursday, December 13, 2018 3:11 PM > To: mozilla-dev-security-policy > > Subject: Re: Underscore characters and DigiCert > > There are currently no program requirements for roots that have had their > websites trust bit turned off or been removed from NSS, but this is an open > area of concern [1]. When a root is disabled or removed, there is no > protection for Firefox users who haven't updated to a current version, nor > for any of the other consumers of our root store until they update. > > However, that doesn't apply here. These roots are still in the Mozilla root > store and trusted for TLS, and some of them will be for quite a while due > to > the whitelisted Apple & Google intermediates [2]. It is clear that Mozilla > policy, and in-turn the BRs, still apply to these roots. > > Should DigiCert decide not to revoke certificates containing underscores by > the 15-Jan deadline in SC12, including those chaining to distrusted > Symantec > roots, I plan to treat it as an incident. As with any incident, full > disclosure is the expectation. > > - Wayne > > [1] https://github.com/mozilla/pkipolicy/issues/124 > [2] https://wiki.mozilla.org/CA/Additional_Trust_Changes#Symantec >
RE: Underscore characters and DigiCert
Can we request removal of these roots now? This seems very similar to the SHA1 situation where CAs requested root removal and then treated the root as private, regardless of the trust in older platforms. -Original Message- From: dev-security-policy On Behalf Of Wayne Thayer via dev-security-policy Sent: Thursday, December 13, 2018 3:11 PM To: mozilla-dev-security-policy Subject: Re: Underscore characters and DigiCert There are currently no program requirements for roots that have had their websites trust bit turned off or been removed from NSS, but this is an open area of concern [1]. When a root is disabled or removed, there is no protection for Firefox users who haven't updated to a current version, nor for any of the other consumers of our root store until they update. However, that doesn't apply here. These roots are still in the Mozilla root store and trusted for TLS, and some of them will be for quite a while due to the whitelisted Apple & Google intermediates [2]. It is clear that Mozilla policy, and in-turn the BRs, still apply to these roots. Should DigiCert decide not to revoke certificates containing underscores by the 15-Jan deadline in SC12, including those chaining to distrusted Symantec roots, I plan to treat it as an incident. As with any incident, full disclosure is the expectation. - Wayne [1] https://github.com/mozilla/pkipolicy/issues/124 [2] https://wiki.mozilla.org/CA/Additional_Trust_Changes#Symantec On Wed, Dec 12, 2018 at 5:54 PM Jeremy Rowley via dev-security-policy < dev-security-policy@lists.mozilla.org> wrote: > Hey all, > > We're working towards revoking certs with underscore characters in the > domain name, per SC12, but I had a question about legacy Symantec > systems and Mozilla. These particular roots are no longer trusted for > TLS certs in Google or Mozilla, which means the applicability of the > BRs is dubious. The roots are shortly being removed from Microsoft and > Apple, although that's more of an FYI rather than something with > direct bearing on the Mozilla community. If the roots are no longer > trusted for TLS in Mozilla, is there any requirement to revoke the certs issued under those roots? > > > > My initial thought is no as this is similar to what Comodo did with > their request to remove a SHA1 root (and what DigiCert did with one of > the Verizon roots). Note these are still flagged by zlint because they > are trusted in older systems. Because the situation is slightly > different with the way distrust was technically implemented, I wanted > to see if there were any concerns with the community about treating > these as private going forward, similar to the SHA1 roots. Thoughts? > > > > Jeremy > > ___ dev-security-policy mailing list dev-security-policy@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-security-policy smime.p7s Description: S/MIME cryptographic signature ___ dev-security-policy mailing list dev-security-policy@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-security-policy
Re: Underscore characters and DigiCert
There are currently no program requirements for roots that have had their websites trust bit turned off or been removed from NSS, but this is an open area of concern [1]. When a root is disabled or removed, there is no protection for Firefox users who haven't updated to a current version, nor for any of the other consumers of our root store until they update. However, that doesn't apply here. These roots are still in the Mozilla root store and trusted for TLS, and some of them will be for quite a while due to the whitelisted Apple & Google intermediates [2]. It is clear that Mozilla policy, and in-turn the BRs, still apply to these roots. Should DigiCert decide not to revoke certificates containing underscores by the 15-Jan deadline in SC12, including those chaining to distrusted Symantec roots, I plan to treat it as an incident. As with any incident, full disclosure is the expectation. - Wayne [1] https://github.com/mozilla/pkipolicy/issues/124 [2] https://wiki.mozilla.org/CA/Additional_Trust_Changes#Symantec On Wed, Dec 12, 2018 at 5:54 PM Jeremy Rowley via dev-security-policy < dev-security-policy@lists.mozilla.org> wrote: > Hey all, > > We're working towards revoking certs with underscore characters in the > domain name, per SC12, but I had a question about legacy Symantec systems > and Mozilla. These particular roots are no longer trusted for TLS certs in > Google or Mozilla, which means the applicability of the BRs is dubious. The > roots are shortly being removed from Microsoft and Apple, although that's > more of an FYI rather than something with direct bearing on the Mozilla > community. If the roots are no longer trusted for TLS in Mozilla, is there > any requirement to revoke the certs issued under those roots? > > > > My initial thought is no as this is similar to what Comodo did with their > request to remove a SHA1 root (and what DigiCert did with one of the > Verizon > roots). Note these are still flagged by zlint because they are trusted in > older systems. Because the situation is slightly different with the way > distrust was technically implemented, I wanted to see if there were any > concerns with the community about treating these as private going forward, > similar to the SHA1 roots. Thoughts? > > > > Jeremy > > ___ dev-security-policy mailing list dev-security-policy@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-security-policy
Underscore characters and DigiCert
Hey all, We're working towards revoking certs with underscore characters in the domain name, per SC12, but I had a question about legacy Symantec systems and Mozilla. These particular roots are no longer trusted for TLS certs in Google or Mozilla, which means the applicability of the BRs is dubious. The roots are shortly being removed from Microsoft and Apple, although that's more of an FYI rather than something with direct bearing on the Mozilla community. If the roots are no longer trusted for TLS in Mozilla, is there any requirement to revoke the certs issued under those roots? My initial thought is no as this is similar to what Comodo did with their request to remove a SHA1 root (and what DigiCert did with one of the Verizon roots). Note these are still flagged by zlint because they are trusted in older systems. Because the situation is slightly different with the way distrust was technically implemented, I wanted to see if there were any concerns with the community about treating these as private going forward, similar to the SHA1 roots. Thoughts? Jeremy smime.p7s Description: S/MIME cryptographic signature ___ dev-security-policy mailing list dev-security-policy@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-security-policy