Re: Apple: Patch Management

2019-12-13 Thread Apple CA via dev-security-policy
On Monday, December 9, 2019 at 2:03:20 PM UTC-8, Matt Palmer wrote:
> On Fri, Dec 06, 2019 at 07:08:46PM -0800, Apple CA via dev-security-policy 
> wrote:
> > On Saturday, November 23, 2019 at 3:28:10 PM UTC-8, Matt Palmer wrote:
> > > [aside: this is how incident reports should be done, IMHO]
> > > 
> > > On Fri, Nov 22, 2019 at 07:23:27PM -0800, Apple CA via 
> > > dev-security-policy wrote:
> > > > We did not have an accurate understanding of how the vulnerability 
> > > > scanner
> > > > worked.  Our understanding of its capabilities lead us to believe it was
> > > > scanning and detecting vulnerabilities in EJBCA.
> > > 
> > > There's a reasonable chance that other CAs may have a similar situation, 
> > > so
> > > I think it's worth digging deeper into the root causes here.  Can you 
> > > expand
> > > on how this misunderstanding regarding the vulnerability scanner came to
> > > pass?  What was the information on which you were relying when you came to
> > > the understanding of the vulnerability scanner's capabilities?  Were you
> > > misled by the vendor marketing or technical documentation, or was it an
> > > Apple-internal assessment that came to an inaccurate conclution?  Or
> > > "other"?
> > 
> > In order to identify vulnerabilities, the vulnerability scanner (1)
> > attempts to identify/profile software listening on ports and (2) compares
> > software versions against public CVEs and proprietary data sources.  EJBCA
> > is not broadly used software, and the vulnerability scanner did not have
> > custom EJBCA detection logic.  Upon our deeper investigation, we
> > discovered that it (1) only scans the HTTP service and not the EJBCA
> > software, which we would consider insufficient on its own and (2) is not
> > as effective at flagging vulnerabilities in EJBCA because CVEs are not
> > published by EJBCA.  We don’t feel we were mislead by the vendor.
> 
> Thanks for clarifying how the security scanning software worked; that's
> useful.  I'm not confident that we've determined any root causes for the
> failure, though, especially things that other CAs in the ecosystem can learn
> from.  I'll try a different phrasing, which will hopefully provide more
> clarity as to what I'm trying to achieve:
> 
> What specific, actionable items would you recommend all CAs undertake to
> remove or mitigate the risk of this, or a substantially similar, problem
> occurring in their environment?
> 
> > CVEs are not published by EJBCA.
> 
> Does anyone else feel that this is a really, really, *really* bad idea?  The
> CVE system, whilst far from perfect, seems to be the agreed upon medium for
> managing these types of issues, and it disappoints me that EJBCA would
> appear to be "opting out" of it.  Should discussions with EJBCA, either from
> their customers or the wider community, be initiated?
> 
> - Matt

The actions we took are as follows:
* We revisited the functionality of the vulnerability scanner, its ability to 
identify vulnerabilities in the CA software, and questioned our previous 
assumptions about how it functions.
* We revisited the support and patch management notification mechanisms 
employed by the CA software vendor.
* We revisited the Network and Certificate System Security Requirements to 
ensure appropriate coverage for the relevant requirements.
* We no longer rely on the vulnerability scanner as a key control to detect 
security vulnerabilities with EJBCA.
* We made our review of software releases and decisions about upgrading a key 
control.
___
dev-security-policy mailing list
dev-security-policy@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-security-policy


Re: Apple: Patch Management

2019-12-06 Thread Apple CA via dev-security-policy
On Monday, November 25, 2019 at 5:32:12 PM UTC-8, Apple CA wrote:
> On Saturday, November 23, 2019 at 3:28:10 PM UTC-8, Matt Palmer wrote:
> > [aside: this is how incident reports should be done, IMHO]
> > 
> > On Fri, Nov 22, 2019 at 07:23:27PM -0800, Apple CA via dev-security-policy 
> > wrote:
> > > We did not have an accurate understanding of how the vulnerability scanner
> > > worked.  Our understanding of its capabilities lead us to believe it was
> > > scanning and detecting vulnerabilities in EJBCA.
> > 
> > There's a reasonable chance that other CAs may have a similar situation, so
> > I think it's worth digging deeper into the root causes here.  Can you expand
> > on how this misunderstanding regarding the vulnerability scanner came to
> > pass?  What was the information on which you were relying when you came to
> > the understanding of the vulnerability scanner's capabilities?  Were you
> > misled by the vendor marketing or technical documentation, or was it an
> > Apple-internal assessment that came to an inaccurate conclution?  Or
> > "other"?
> > 
> > - Matt
> 
> Thank you for your questions.  Due to the Thanksgiving holiday in the US, we 
> expect to reply to your questions as early as the week of 02 December.

In order to identify vulnerabilities, the vulnerability scanner (1) attempts to 
identify/profile software listening on ports and (2) compares software versions 
against public CVEs and proprietary data sources. EJBCA is not broadly used 
software, and the vulnerability scanner did not have custom EJBCA detection 
logic. Upon our deeper investigation, we discovered that it (1) only scans the 
HTTP service and not the EJBCA software, which we would consider insufficient 
on its own and (2) is not as effective at flagging vulnerabilities in EJBCA 
because CVEs are not published by EJBCA. We don’t feel we were mislead by the 
vendor.
___
dev-security-policy mailing list
dev-security-policy@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-security-policy


Re: Apple: Patch Management

2019-11-25 Thread Apple CA via dev-security-policy
On Saturday, November 23, 2019 at 3:28:10 PM UTC-8, Matt Palmer wrote:
> [aside: this is how incident reports should be done, IMHO]
> 
> On Fri, Nov 22, 2019 at 07:23:27PM -0800, Apple CA via dev-security-policy 
> wrote:
> > We did not have an accurate understanding of how the vulnerability scanner
> > worked.  Our understanding of its capabilities lead us to believe it was
> > scanning and detecting vulnerabilities in EJBCA.
> 
> There's a reasonable chance that other CAs may have a similar situation, so
> I think it's worth digging deeper into the root causes here.  Can you expand
> on how this misunderstanding regarding the vulnerability scanner came to
> pass?  What was the information on which you were relying when you came to
> the understanding of the vulnerability scanner's capabilities?  Were you
> misled by the vendor marketing or technical documentation, or was it an
> Apple-internal assessment that came to an inaccurate conclution?  Or
> "other"?
> 
> - Matt

Thank you for your questions.  Due to the Thanksgiving holiday in the US, we 
expect to reply to your questions as early as the week of 02 December.
___
dev-security-policy mailing list
dev-security-policy@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-security-policy


Apple: Patch Management

2019-11-22 Thread Apple CA via dev-security-policy
On November 22, Apple submitted an incident report: 
https://bugzilla.mozilla.org/show_bug.cgi?id=1598829, which is reposted below.

Incident Report

1. How your CA first became aware of the problem (e.g. via a problem report 
submitted to your Problem Reporting Mechanism, a discussion in 
mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the 
time and date.

On 13-November-2018 at 16:00 PT, we completed a more in-depth investigation 
into our key control used to determine if security patches should be applied 
(automated vulnerability scans) and determined that while it was possible the 
vulnerability scanner would detect a vulnerability with EJBCA, it was not 
likely. We also identified that since our review of EJBCA releases and 
decisions about upgrading was not a key control, it was not provided to the 
auditors. Based on this information we committed to opening this incident 
report in Comment 13 of the Apple OCSP responders return responses with 
incorrect issuer incident report. For clarity, key controls are mapped to 
WebTrust control objectives, and non-key controls are performed operationally 
but not provided to the auditors.

2. A timeline of the actions your CA took in response. A timeline is a 
date-and-time-stamped sequence of all relevant events. This may include events 
before the incident was reported, such as when a particular requirement became 
applicable, or a document changed, or a bug was introduced, or an audit was 
done.

14-October-2019 at 15:08 PT - We internally contemplated whether running 
version 4.0.14 could be a potential Network and Certificate System Security 
Requirements (NCSSR) violation, but based on our understanding at the time, we 
determined it was not a violation.

17-October-2019 at 18:33 PT - We posted the Apple OCSP responders return 
responses with incorrect issuer incident report.

17-October-2019 at 19:20 PT - We received Comment #5 questioning the version of 
software we were running on our OCSP service.

21-October-2019 at 9:00 PT - As part of our ongoing investigation related to 
the Apple OCSP responders return responses with incorrect issuer report we took 
a closer look at our software management practices and the CA/Browser Forum 
NCSSR which requires that CAs “Apply recommended security patches to 
Certificate Systems within six (6) months of the security patch’s availability, 
unless the CA documents that the security patch would introduce additional 
vulnerabilities or instabilities that outweigh the benefits of applying the 
security patch.”.

24-October-2019 at 21:54 PT - As part of Comment 6 of Apple OCSP responders 
return responses with incorrect issuer, we committed to opening a separate bug 
with more details.

29-October-2019 from 11:25 - 11:58 PT - Notified the Apple Policy Authority, 
DigiCert and Sectigo (Root vendors), Apple, Microsoft, Mozilla, Google, and 
Oracle (Root programs), and Ernst & Young (WebTrust assessors) of a potential 
incident.

31-October-2019 at 21:36 PT - We posted Comment 10 to the Apple OCSP responders 
return responses with incorrect issuer incident report in which we stated that 
our practices and controls for both the OCSP software and CA software were 
compliant with the NCSSR.

31-October-2019 from 21:38 - 21:40 PT - Notified the Apple Policy Authority, 
DigiCert and Sectigo (Root vendors), Apple, Microsoft, Mozilla, Google, and 
Oracle (Root programs), and Ernst & Young (WebTrust assessors).

13-November-2018 at 16:00 PT - We completed a more in-depth investigation into 
our key control used to determine if security patches should be applied and 
determined that while it was possible the vulnerability scanner would detect a 
vulnerability with EJBCA, it was not likely. We also identified that since our 
review of EJBCA releases and decisions about upgrading was not a key control, 
it was not provided to the auditors. Therefore, we came to the conclusion that 
our key control does not go far enough to meet the requirements specified in 1l 
of the NCSSR. Based on this information, we committed to opening this incident 
report in Comment 13 with more details.

14-November-2019 from 14:51 - 14:53 PT - Notified the Apple Policy Authority, 
DigiCert and Sectigo (Root vendors), Apple, Microsoft, Mozilla, Google, and 
Oracle (Root programs), and Ernst & Young (WebTrust assessors).

14-November-2019 at 16:00 PT - We made our review of software releases and 
decisions about upgrading a key control.

3. Whether your CA has stopped, or has not yet stopped, issuing certificates 
with the problem. A statement that you have will be considered a pledge to the 
community; a statement that you have not requires an explanation.

No non-compliant certificates were issued. We made our review of software 
releases and decisions about upgrading a key control.

4. A summary of the problematic certificates. For each problem: number of 
certs, and the date the first and last certs with that problem were 

Re: Apple OCSP responders return responses with incorrect issuer

2019-10-17 Thread Apple CA via dev-security-policy
On October 17, Apple submitted an incident report: 
https://bugzilla.mozilla.org/show_bug.cgi?id=1588001#c3, which is reposted 
below. 

Incident Report

1. How your CA first became aware of the problem (e.g. via a problem report 
submitted to your Problem Reporting Mechanism, a discussion in 
mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the 
time and date.

On 03-October-2019 at 13:51 PT, we were notified via a problem report submitted 
to our Problem Reporting Mechanism that our OCSP responders were returning 
signed responses with incorrect issuer. 

2. A timeline of the actions your CA took in response. A timeline is a 
date-and-time-stamped sequence of all relevant events. This may include events 
before the incident was reported, such as when a particular requirement became 
applicable, or a document changed, or a bug was introduced, or an audit was 
done.

03-October-2019 at 13:51 PT - We were notified via a problem report submitted 
to our Problem Reporting Mechanism that our OCSP responders were returning 
signed responses with incorrect issuer.

03-October-2019 at 13:52 PT - The compliance team read the problem report and 
began pulling the appropriate people together to begin the investigation.

03-October-2019 at 14:15 PT - Investigation began to confirm the reported 
behavior. 

03-October-2019 at 16:15 PT -  Based on an initial investigation, we determined 
that in some cases when the OCSP service receives a request it can’t process, 
it returns a status of unknown signed with a default OCSP responder, which is 
not always signed by the CA that issued the certificate whose revocation status 
is being checked.

03-October-2019 at 19:21 - Notified DigiCert (Root vendor).

03-October-2019 at 20:10 - We responded to the reporter with an initial 
acknowledgement and a commitment to investigate further and respond with more 
details within 24 hours of the report being submitted.

04-October-2019 at 10:10 PT - Notified Sectigo (Root vendor).

04-October-2019 at 11:26 PT - Notified Ernst & Young (WebTrust assessors).

04-October-2019 at 13:48 PT - We provided a preliminary report on our findings 
to the individual who filed the problem report.

07-October-2019 at 14:15 PT - We began rolling out a fix to our OCSP service. 
The fix was to disable the default OCSP responder so that the responses are 
always signed by the CA that issued the certificate whose revocation status is 
being checked. Disabling the default OCSP responder ensures that the responder 
will reply ‘unauthorized’  (as per RFC 6960) for all unknown issuers. The 
issuer may be unknown if the OCSP service cannot identify the issuer, such as 
when an OCSP client uses a hash algorithm for CertID that the OCSP service does 
not support or when the request indicates an unrecognized issuer that is not 
served by our OCSP service.

10-October-2019 at 19:38 PT  - Posted initial incident report to Bugzilla.

17-October-2019 - The fix has been pushed out to the majority of our production 
OCSP service and scheduled for completion by 18-October-2019.

3. Whether your CA has stopped, or has not yet stopped, issuing certificates 
with the problem. A statement that you have will be considered a pledge to the 
community; a statement that you have not requires an explanation.

No non-compliant certificates were issued. The OCSP fix has been pushed out to 
the majority of our production OCSP service and scheduled for completion by 
18-October-2019.

4. A summary of the problematic certificates. For each problem: number of 
certs, and the date the first and last certs with that problem were issued.

No non-compliant certificates were issued.

5. The complete certificate data for the problematic certificates. The 
recommended way to provide this is to ensure each certificate is logged to CT 
and then list the fingerprints or crt.sh IDs, either in the report or as an 
attached spreadsheet, with one list per distinct problem.

No non-compliant certificates were issued.

6. Explanation about how and why the mistakes were made or bugs introduced, and 
how they avoided detection until now.

When the OCSP service was first set up in 2012, the OCSP service software did 
not allow the default OCSP responder to be disabled. We began issuing publicly 
trusted TLS certificates in 2014 and the default OCSP responder was not 
disabled. We did not identify this as an issue because our test cases did not 
address scenarios in which our OCSP service could not identify the issuer and 
thus signs the response with the default OCSP responder.

The default OCSP responder is only used when the OCSP service cannot identify 
the issuer. This may occur when an OCSP client uses a hash algorithm for CertID 
that the OCSP service does not support or when the request indicates an 
unrecognized issuer that is not served by our OCSP service. 

We were aware of section 4.9.9 of the Baseline Requirements that states that 
OCSP responses must “Be signed by 

Apple OCSP responders return responses with incorrect issuer

2019-10-10 Thread Apple CA via dev-security-policy
Apple has submitted this preliminary incident report: 
https://bugzilla.mozilla.org/show_bug.cgi?id=1588001, which is reposted below.  
 

On 03-October-2019 at 13:52 PT, we were notified via a problem report submitted 
to our Problem Reporting Mechanism that our OCSP responders were returning 
signed responses with incorrect issuer. Based on an initial investigation, 
we’ve determined that in some cases when the OCSP service receives a request it 
cannot process, it signs the response with a default OCSP responder (our OCSP 
service processes requests for multiple CAs). We are investigating a fix so 
that responses are not signed by an incorrect OCSP responder.

Further details will be provided no later than 17-October-2019.
___
dev-security-policy mailing list
dev-security-policy@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-security-policy


Apple: Precertificates without corresponding certificates return OCSP value of "unknown"

2019-09-13 Thread Apple CA via dev-security-policy
We’ve been following the discussions regarding how OCSP responders should 
handle Precertificates without corresponding certificates and what the 
appropriate response indicator should be (good, revoked, or unknown). 

Based on the recent clarifications at [1], we want to inform the community that 
Apple’s OCSP responders return a status of “unknown” for Precertificates 
without a corresponding certificate. We have identified one Precertificate that 
did not result in a corresponding certificate for which our OCSP responders are 
returning a status of “unknown” (https://crt.sh/?id=1368484681).

We’ve updated the OCSP responders to respond “good” for that Precertificate and 
a long-term fix is in progress.

We appreciate the efforts being made to amend the Mozilla Root Store Policy to 
explicitly address matters relating to Certificate Transparency.

[1] 
https://groups.google.com/d/msg/mozilla.dev.security.policy/LC_y8yPDI9Q/24Fl9kc-AQAJ
___
dev-security-policy mailing list
dev-security-policy@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-security-policy


Apple: Non-compliant Common Name Length

2019-06-05 Thread Apple CA via dev-security-policy
On June 4, Apple submitted an incident report: 
https://bugzilla.mozilla.org/show_bug.cgi?id=1556906, which is reposted below.
___
 

Incident Report

1. How your CA first became aware of the problem (e.g. via a problem report 
submitted to your Problem Reporting Mechanism, a discussion in 
mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the 
time and date.

On 2019-05-21 14:30 PT, the CA compliance team was notified by an internal 
developer that during the course of a code review, it was discovered that 
certificates had been issued with Common Names (CNs) longer than 64 characters. 

2. A timeline of the actions your CA took in response. A timeline is a 
date-and-time-stamped sequence of all relevant events. This may include events 
before the incident was reported, such as when a particular requirement became 
applicable, or a document changed, or a bug was introduced, or an audit was 
done.

2019-05-15 9:00 PT - Code review identified that the software that checks 
certificates for Baseline compliance was not enforcing a max length of 64 
characters for CNs.
2019-05-15 11:24 PT - The only two impacted certificates that were still valid 
were revoked by the developer who identified the issue.
2019-05-21 14:30 PT - Compliance team was notified about the issue.
2019-05-21 18:00 PT -  Risk assessment was completed. 
2019-05-22: 10:00 PT - Software fix was deployed to the production environment.
2019-05-23: 8:42 PT - Requested meeting with DigiCert (the Root CA) to discuss 
the incident.
2019-05-24: 13:00 PT - Notified Ernst & Young (WebTrust assessors).

3. Whether your CA has stopped, or has not yet stopped, issuing certificates 
with the problem. A statement that you have will be considered a pledge to the 
community; a statement that you have not requires an explanation.

A software update that prevents issuance of certificates with CNs longer than 
64 characters was deployed in production on 2019-05-22 at 10:00 PT.

4. A summary of the problematic certificates. For each problem: number of 
certs, and the date the first and last certs with that problem were issued.

28 certificates were impacted between 2014-11-28 and 2019-03-25.

5. The complete certificate data for the problematic certificates. The 
recommended way to provide this is to ensure each certificate is logged to CT 
and then list the fingerprints or crt.sh IDs, either in the report or as an 
attached spreadsheet, with one list per distinct problem.

A file has been attached with a list of all impacted certificates.

6. Explanation about how and why the mistakes were made or bugs introduced, and 
how they avoided detection until now.

The software that checks certificates for Baseline compliance prior to issuance 
and for quarterly self audits was not enforcing a max length of 64 characters 
for the CN.

7. List of steps your CA is taking to resolve the situation and ensure such 
issuance will not be repeated in the future, accompanied with a timeline of 
when your CA expects to accomplish these things.

i.  A software fix that enforces a maximum of 64 character CNs was deployed in 
production on May 22nd. 
ii.  The internal notification process will be enhanced by mid-June to minimize 
the time between identification of a suspected issue and communication to the 
compliance team.
iii.  We plan to implement a second linter (most likely zLint) by end of June, 
which is based on a separate code base, to strengthen the ability to prevent 
and detect mis-issuance.
___
dev-security-policy mailing list
dev-security-policy@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-security-policy