Hi all, I'm sure questions of certificates leaked to the public via GitHub and other file sharing / code sharing / deployment repository hosting and sharing sites have come up before, but last night I spent a couple of hours constructing various search criteria which I don't think were even especially clever, but still I was shocked and amazed at what I found:
At least 10 different Apple Development and Production environment certificates and key pairs. Of these minimum of 10 that I am counting, I validated that the certificate is within its validity period. Of these, I validated that the key I found matches the public key information in the certificate. Most of these certificates are TLS Client Authentication certificates which also have additional Apple proprietary extended key usages. These certificates are utilized for authenticating to the Apple Push Notification System. A couple of certificates were Apple Developer ID certificates appropriate for development and production environment deployment of executable code to Apple devices. (Those developer ID certificates I have reported to Apple for revocation.) There were more Apple Push authentication certificates than I cared to write up and send over. I was shocked at the level of improper distribution and leaking of these keys and certificates. Once in a while, the key was represented in encrypted form. In _every_ instance for which I found an encrypted key and dug further, either a piece of code, a configuration file, or sometimes a README-KEY-PASSWORD.txt (or similar) within the same repository successfully decrypted the encrypted key. Additionally, I did find some TLS server certificates. There were many more that I did not bother to carefully analyze. Some were expired. One was a in-validity-window DV certificate issued by Let's Encrypt. Utilizing the certificate's private key, I was able to successfully use the Let's Encrypt ACME API to automatically request revocation of that certificate. Minutes later, I verified that OCSP responses for that certificate were, in fact, indicating that the certificate was revoked. Of course, revocation even with a really nice OCSP responder system is not very effective today. I have this suspicion that human nature dictates that eliminating these kinds of key material leaks is not even a goal worth having. Disappointment, I suspect, lives down that road. Because live OCSP checks for certificates en-masse is not appealing to either the CAs or the browsers or the end users (consequences of network delay, reliability, etc.), revocation means very little pragmatically today. This only reinforces the value and importance of either/both: - Quite short lived certificates, automatically replaced and deployed, to reduce the risks associated with key compromise and/or - OCSP must-staple, which I believe is only pragmatically gated at the moment by a number of really poor server-side implementations of OCSP stapling. Servers must cache good responses. Servers must use those while awaiting a new good response further into the OCSP response validity period. Servers must validate the response and not server random garbage as if OCSP. Etc, etc. Ryan Sleevi's work documenting the core issues is clearly a step in the right direction. Both NGINX's and Apache HTTPD's implementations of OCSP stapling are lacking in several material respects. It would certainly be a significant undertaking, but I believe that organizations who are working to ensure a secure Web (and that reap the benefits of a secure and trustworthy web) could do much to achieve better deployment of OCSP stapling in relatively short time: 1. Direct contribution of funds / bounty to the core developers of each of those two web server projects for building a server-side OCSP stapling implementation which is trivial to configure and which meets the needs of an ideal implementation with respect to caching of good results, validating new responses to staple, scheduling the deployment of successful new responses or scheduling retries of fails, etc. Insist that the code be written with a view to maximal back-port capability for said implementations. 2. If such contributions are infeasible, funding competent external development of code which achieves the same as item 1 above. 3. High level engagement with major distributions. Tackle the technical and administrative hurdles to get these changes into the stable and development builds of all currently shipping versions of at least RedHat's, Canonical's, and Debian's distributions. Get these changes into the standard default version httpd and nginx updates. 4. Same as above but for common docker images, prevalent VM images, etc. 5. Ensure that the browsers are ready to support and enforce fail-hard on certificates which feature the OCSP must-staple extension. 6. Monitor progress in readiness and incentivize deployment of OCSP must-staple at the operations level. Maybe CAs are able to / encouraged to offer longer validity periods on OCSP must-staple certificates? In terms of dollars and cents, I think the various giant organizations (Hi, Google, how are you today?) most interested and best positioned to benefit from a secure web would regard the costs of a coordinated effort such as this pretty insubstantial. Through a little Google digging, I find numerous comments and references from well informed parties going back quite several years lamenting the poor state of support of OCSP stapling in both Apache HTTPD and NGINX. I'm well aware of the rising power that is Caddy, but it's not there yet. The whole ecosystem could be greatly helped by making the default shipping versions of those two daemons in the major distros be ideal OCSP-stapling ready. Just a thought... Matt Hardeman _______________________________________________ dev-security-policy mailing list [email protected] https://lists.mozilla.org/listinfo/dev-security-policy
