Re: Apple: Patch Management
On Monday, December 9, 2019 at 2:03:20 PM UTC-8, Matt Palmer wrote: > On Fri, Dec 06, 2019 at 07:08:46PM -0800, Apple CA via dev-security-policy > wrote: > > On Saturday, November 23, 2019 at 3:28:10 PM UTC-8, Matt Palmer wrote: > > > [aside: this is how incident reports should be done, IMHO] > > > > > > On Fri, Nov 22, 2019 at 07:23:27PM -0800, Apple CA via > > > dev-security-policy wrote: > > > > We did not have an accurate understanding of how the vulnerability > > > > scanner > > > > worked. Our understanding of its capabilities lead us to believe it was > > > > scanning and detecting vulnerabilities in EJBCA. > > > > > > There's a reasonable chance that other CAs may have a similar situation, > > > so > > > I think it's worth digging deeper into the root causes here. Can you > > > expand > > > on how this misunderstanding regarding the vulnerability scanner came to > > > pass? What was the information on which you were relying when you came to > > > the understanding of the vulnerability scanner's capabilities? Were you > > > misled by the vendor marketing or technical documentation, or was it an > > > Apple-internal assessment that came to an inaccurate conclution? Or > > > "other"? > > > > In order to identify vulnerabilities, the vulnerability scanner (1) > > attempts to identify/profile software listening on ports and (2) compares > > software versions against public CVEs and proprietary data sources. EJBCA > > is not broadly used software, and the vulnerability scanner did not have > > custom EJBCA detection logic. Upon our deeper investigation, we > > discovered that it (1) only scans the HTTP service and not the EJBCA > > software, which we would consider insufficient on its own and (2) is not > > as effective at flagging vulnerabilities in EJBCA because CVEs are not > > published by EJBCA. We don’t feel we were mislead by the vendor. > > Thanks for clarifying how the security scanning software worked; that's > useful. I'm not confident that we've determined any root causes for the > failure, though, especially things that other CAs in the ecosystem can learn > from. I'll try a different phrasing, which will hopefully provide more > clarity as to what I'm trying to achieve: > > What specific, actionable items would you recommend all CAs undertake to > remove or mitigate the risk of this, or a substantially similar, problem > occurring in their environment? > > > CVEs are not published by EJBCA. > > Does anyone else feel that this is a really, really, *really* bad idea? The > CVE system, whilst far from perfect, seems to be the agreed upon medium for > managing these types of issues, and it disappoints me that EJBCA would > appear to be "opting out" of it. Should discussions with EJBCA, either from > their customers or the wider community, be initiated? > > - Matt The actions we took are as follows: * We revisited the functionality of the vulnerability scanner, its ability to identify vulnerabilities in the CA software, and questioned our previous assumptions about how it functions. * We revisited the support and patch management notification mechanisms employed by the CA software vendor. * We revisited the Network and Certificate System Security Requirements to ensure appropriate coverage for the relevant requirements. * We no longer rely on the vulnerability scanner as a key control to detect security vulnerabilities with EJBCA. * We made our review of software releases and decisions about upgrading a key control. ___ dev-security-policy mailing list dev-security-policy@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-security-policy
Re: Apple: Patch Management
On Fri, Dec 06, 2019 at 07:08:46PM -0800, Apple CA via dev-security-policy wrote: > On Saturday, November 23, 2019 at 3:28:10 PM UTC-8, Matt Palmer wrote: > > [aside: this is how incident reports should be done, IMHO] > > > > On Fri, Nov 22, 2019 at 07:23:27PM -0800, Apple CA via dev-security-policy > > wrote: > > > We did not have an accurate understanding of how the vulnerability scanner > > > worked. Our understanding of its capabilities lead us to believe it was > > > scanning and detecting vulnerabilities in EJBCA. > > > > There's a reasonable chance that other CAs may have a similar situation, so > > I think it's worth digging deeper into the root causes here. Can you expand > > on how this misunderstanding regarding the vulnerability scanner came to > > pass? What was the information on which you were relying when you came to > > the understanding of the vulnerability scanner's capabilities? Were you > > misled by the vendor marketing or technical documentation, or was it an > > Apple-internal assessment that came to an inaccurate conclution? Or > > "other"? > > In order to identify vulnerabilities, the vulnerability scanner (1) > attempts to identify/profile software listening on ports and (2) compares > software versions against public CVEs and proprietary data sources. EJBCA > is not broadly used software, and the vulnerability scanner did not have > custom EJBCA detection logic. Upon our deeper investigation, we > discovered that it (1) only scans the HTTP service and not the EJBCA > software, which we would consider insufficient on its own and (2) is not > as effective at flagging vulnerabilities in EJBCA because CVEs are not > published by EJBCA. We don’t feel we were mislead by the vendor. Thanks for clarifying how the security scanning software worked; that's useful. I'm not confident that we've determined any root causes for the failure, though, especially things that other CAs in the ecosystem can learn from. I'll try a different phrasing, which will hopefully provide more clarity as to what I'm trying to achieve: What specific, actionable items would you recommend all CAs undertake to remove or mitigate the risk of this, or a substantially similar, problem occurring in their environment? > CVEs are not published by EJBCA. Does anyone else feel that this is a really, really, *really* bad idea? The CVE system, whilst far from perfect, seems to be the agreed upon medium for managing these types of issues, and it disappoints me that EJBCA would appear to be "opting out" of it. Should discussions with EJBCA, either from their customers or the wider community, be initiated? - Matt ___ dev-security-policy mailing list dev-security-policy@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-security-policy
Re: Apple: Patch Management
On Monday, November 25, 2019 at 5:32:12 PM UTC-8, Apple CA wrote: > On Saturday, November 23, 2019 at 3:28:10 PM UTC-8, Matt Palmer wrote: > > [aside: this is how incident reports should be done, IMHO] > > > > On Fri, Nov 22, 2019 at 07:23:27PM -0800, Apple CA via dev-security-policy > > wrote: > > > We did not have an accurate understanding of how the vulnerability scanner > > > worked. Our understanding of its capabilities lead us to believe it was > > > scanning and detecting vulnerabilities in EJBCA. > > > > There's a reasonable chance that other CAs may have a similar situation, so > > I think it's worth digging deeper into the root causes here. Can you expand > > on how this misunderstanding regarding the vulnerability scanner came to > > pass? What was the information on which you were relying when you came to > > the understanding of the vulnerability scanner's capabilities? Were you > > misled by the vendor marketing or technical documentation, or was it an > > Apple-internal assessment that came to an inaccurate conclution? Or > > "other"? > > > > - Matt > > Thank you for your questions. Due to the Thanksgiving holiday in the US, we > expect to reply to your questions as early as the week of 02 December. In order to identify vulnerabilities, the vulnerability scanner (1) attempts to identify/profile software listening on ports and (2) compares software versions against public CVEs and proprietary data sources. EJBCA is not broadly used software, and the vulnerability scanner did not have custom EJBCA detection logic. Upon our deeper investigation, we discovered that it (1) only scans the HTTP service and not the EJBCA software, which we would consider insufficient on its own and (2) is not as effective at flagging vulnerabilities in EJBCA because CVEs are not published by EJBCA. We don’t feel we were mislead by the vendor. ___ dev-security-policy mailing list dev-security-policy@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-security-policy
Re: Apple: Patch Management
On Saturday, November 23, 2019 at 3:28:10 PM UTC-8, Matt Palmer wrote: > [aside: this is how incident reports should be done, IMHO] > > On Fri, Nov 22, 2019 at 07:23:27PM -0800, Apple CA via dev-security-policy > wrote: > > We did not have an accurate understanding of how the vulnerability scanner > > worked. Our understanding of its capabilities lead us to believe it was > > scanning and detecting vulnerabilities in EJBCA. > > There's a reasonable chance that other CAs may have a similar situation, so > I think it's worth digging deeper into the root causes here. Can you expand > on how this misunderstanding regarding the vulnerability scanner came to > pass? What was the information on which you were relying when you came to > the understanding of the vulnerability scanner's capabilities? Were you > misled by the vendor marketing or technical documentation, or was it an > Apple-internal assessment that came to an inaccurate conclution? Or > "other"? > > - Matt Thank you for your questions. Due to the Thanksgiving holiday in the US, we expect to reply to your questions as early as the week of 02 December. ___ dev-security-policy mailing list dev-security-policy@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-security-policy
Re: Apple: Patch Management
[aside: this is how incident reports should be done, IMHO] On Fri, Nov 22, 2019 at 07:23:27PM -0800, Apple CA via dev-security-policy wrote: > We did not have an accurate understanding of how the vulnerability scanner > worked. Our understanding of its capabilities lead us to believe it was > scanning and detecting vulnerabilities in EJBCA. There's a reasonable chance that other CAs may have a similar situation, so I think it's worth digging deeper into the root causes here. Can you expand on how this misunderstanding regarding the vulnerability scanner came to pass? What was the information on which you were relying when you came to the understanding of the vulnerability scanner's capabilities? Were you misled by the vendor marketing or technical documentation, or was it an Apple-internal assessment that came to an inaccurate conclution? Or "other"? - Matt ___ dev-security-policy mailing list dev-security-policy@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-security-policy
Apple: Patch Management
On November 22, Apple submitted an incident report: https://bugzilla.mozilla.org/show_bug.cgi?id=1598829, which is reposted below. Incident Report 1. How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date. On 13-November-2018 at 16:00 PT, we completed a more in-depth investigation into our key control used to determine if security patches should be applied (automated vulnerability scans) and determined that while it was possible the vulnerability scanner would detect a vulnerability with EJBCA, it was not likely. We also identified that since our review of EJBCA releases and decisions about upgrading was not a key control, it was not provided to the auditors. Based on this information we committed to opening this incident report in Comment 13 of the Apple OCSP responders return responses with incorrect issuer incident report. For clarity, key controls are mapped to WebTrust control objectives, and non-key controls are performed operationally but not provided to the auditors. 2. A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done. 14-October-2019 at 15:08 PT - We internally contemplated whether running version 4.0.14 could be a potential Network and Certificate System Security Requirements (NCSSR) violation, but based on our understanding at the time, we determined it was not a violation. 17-October-2019 at 18:33 PT - We posted the Apple OCSP responders return responses with incorrect issuer incident report. 17-October-2019 at 19:20 PT - We received Comment #5 questioning the version of software we were running on our OCSP service. 21-October-2019 at 9:00 PT - As part of our ongoing investigation related to the Apple OCSP responders return responses with incorrect issuer report we took a closer look at our software management practices and the CA/Browser Forum NCSSR which requires that CAs “Apply recommended security patches to Certificate Systems within six (6) months of the security patch’s availability, unless the CA documents that the security patch would introduce additional vulnerabilities or instabilities that outweigh the benefits of applying the security patch.”. 24-October-2019 at 21:54 PT - As part of Comment 6 of Apple OCSP responders return responses with incorrect issuer, we committed to opening a separate bug with more details. 29-October-2019 from 11:25 - 11:58 PT - Notified the Apple Policy Authority, DigiCert and Sectigo (Root vendors), Apple, Microsoft, Mozilla, Google, and Oracle (Root programs), and Ernst & Young (WebTrust assessors) of a potential incident. 31-October-2019 at 21:36 PT - We posted Comment 10 to the Apple OCSP responders return responses with incorrect issuer incident report in which we stated that our practices and controls for both the OCSP software and CA software were compliant with the NCSSR. 31-October-2019 from 21:38 - 21:40 PT - Notified the Apple Policy Authority, DigiCert and Sectigo (Root vendors), Apple, Microsoft, Mozilla, Google, and Oracle (Root programs), and Ernst & Young (WebTrust assessors). 13-November-2018 at 16:00 PT - We completed a more in-depth investigation into our key control used to determine if security patches should be applied and determined that while it was possible the vulnerability scanner would detect a vulnerability with EJBCA, it was not likely. We also identified that since our review of EJBCA releases and decisions about upgrading was not a key control, it was not provided to the auditors. Therefore, we came to the conclusion that our key control does not go far enough to meet the requirements specified in 1l of the NCSSR. Based on this information, we committed to opening this incident report in Comment 13 with more details. 14-November-2019 from 14:51 - 14:53 PT - Notified the Apple Policy Authority, DigiCert and Sectigo (Root vendors), Apple, Microsoft, Mozilla, Google, and Oracle (Root programs), and Ernst & Young (WebTrust assessors). 14-November-2019 at 16:00 PT - We made our review of software releases and decisions about upgrading a key control. 3. Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem. A statement that you have will be considered a pledge to the community; a statement that you have not requires an explanation. No non-compliant certificates were issued. We made our review of software releases and decisions about upgrading a key control. 4. A summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were
