Hi everyone,  


This email is intended to gather public and browser feedback on how we are
handling the transitioning Verizon's customers to DigiCert and share with
everyone the plan for when all non-DigiCert hosted sub CAs will be fully
compliant with the BRs and network security guidelines. Primarily, this
letter addresses when all issuing CAs previously held by Verizon will be
covered by an unqualified audit and how we are responding to the sub CAs
that issued SHA1 certificates. We are looking forward to the browser and
public feedback on how to handle the non-compliant cross-signs and insight
on how the public views our transition progress and planned completion



Prior to presenting the plan for remediation, I thought I'd share a bit
about our progress with migrating Verizon customers.  About a year ago in
July, DigiCert closed an agreement with Verizon where DigiCert took
ownership of three publicly-trusted Verizon root certificates. These
certificates are the Verizon Global Root CA, the Baltimore CyberTrust Root
CA, and the Verizon Root CA. There were many reasons the acquisition made
sense, including acquisition of a root that had cross-signed DigiCert for
many years. The ubiquity of the Verizon roots is wide-spread, which meant a
lot of CAs like to cross-sign with them. Another significant reason for the
acquisition was an attempt to improve the CA industry. After watching the
issuance of wildcard EV certs, non-compliant subordinate CAs, and certs with
poor profiles, we made a conscious decision to purchase these roots with the
intention of migrating them to more complaint system. We felt that helping
these CAs get to the point of regular audits and best practices would raise
the quality of the entire industry. 


Prior to the acquisition, we were made aware of several potential
non-compliances by Verizon's customers. We worked closely with Verizon to
clean up many of the Sub CAs, including revoking CAs that would never be
compliant with the requirements and attempting to technically constrain
others.  Sub CAs were revoked because they either didn't have an audit, were
unresponsive to communication, or couldn't control their issuance. Verizon
was a great partner and took a very proactive approach in removing CAs that
were not actively working towards compliance. Unfortunately, the age of the
roots and large number of cross-signs led to a lot of missing paperwork and
certain issues in identifying which CAs were covered by audits. Prior to
closing we believed there were approximately five technically constrained
sub CAs and around 20 unconstrained sub CAs. Turns out there were actually
more than 200, each with various levels of compliance. Most of these Sub CAs
operated under the Baltimore Cybertrust root, which was created in 2000.    


To their credit, Verizon revoked 48 of the issuing CAs prior to DigiCert's
acquisition of the roots. Unfortunately, this left around 150 for our small
team to work through. Although the endeavor was daunting, Ben Wilson and
others worked to gather audit reports, email customers about compliance
dates, monitor certificates issued, and revoke non-compliant customers.
Verizon was very good at assisting us in these efforts. This information is
now recorded in the Mozilla database and continuously updated as the
information changes.   


Transition Process 

After our operational acquisition of the roots (which occurred the last part
of September, 2015), we identified 15 expired issuing CAs, 70 revoked sub
CAs, 131 audited sub CAs, and 36 where the status was unknown. Since then,
we've revoked 25 issuing CAs and assisted others with technical constraints,
exodus from the Omniroot cross-signing program, or obtaining audits. Of the
existing sub CAs, about half remain operational and are either audited or
technically constrained. The other half are either winding down or have been
revoked. All revoked certificates were disclosed to Mozilla via Salesforce
and should be part of OneCRL.


Issuing CAs 

There are only a handful of non-audited sub CAs remaining (see
https://crt.sh/mozilla-disclosures#disclosureincomplete). We have a plan for
each of them that we'd like to share with you. We welcome both browser and
public feedback. This is, of course, simply a proposal on how to finish the
transition and provide transparency into where we are at. We are certainly
willing to modify the plan as needed to further online security and meet the
requirements of the root store operators.  


The seven companies listed below are responsible for the remaining unaudited
sub CAs:  


1.       ABB has three unaudited issuing CAs. ABB didn't undergo an audit
earlier this year on the assumption that their sub CAs were technically
constrained. Unfortunately, the constraints weren't properly implemented, a
fact that escaped notice until Rob Stradling's excellent tool exposed a
deficiency a few months ago in how Verizon issuance process. Although
Verizon restricted domain names and IPv4 IP addresses, the CAs could still
issue for the IPv6 space.  Despite promptly replacing the certificates after
discovering the gap, we haven't revoked the issuing CAs. Right now, ABB is
actively working to transition its servers to the new, properly-constrained
certificates. We expect the transition to complete by the end of December
2016. We are revoking the issuing CAs as the migration completes and expect
all of the ABB non-compliant issuing CAs to be revoked on January 5th.  No
new certs are being issued from the faulty issuing CAs.  


2.     Similar to ABB, Verizon created two technically constrained issuing
CAs for Bechtel. Like ABB, Verizon failed to restrict IPv6 issuance for each
of the issuing CAs. Bechtel is actively migrating the remaining 785
certificates to properly constrained issuing CAs. The migration process will
complete at the end of December, after which the non-conforming issuing CAs
will be revoked. They are included in our January 5th revocation plan. This
will also revoke the three issuing CAs created further down in the chain.   


3.     Dell also thought they were technically constrained. Similar to ABB
and Bechtel, the restriction was inadequate. Fortunately, Dell isn't using
the issuing CAs and doesn't require migration. We've previously revoked four
issuing certificates (Bug 1279066) but missed one during the process. We
plan to immediately revoke the remaining issuing CA and all CAs subordinate
to the issuing CA. 


4.     While Certipost is hosted by Verizon's internal infrastructure, the
Verizon WebTrust audit letter did not specifically identify the Certipost
CAs as audited. DigiCert has been in the process of revoking these CAs but
there is some confusion about the effects of the revocation. Apparently the
certificates are used for air traffic control in Europe and there is a fear
that revoking the intermediate may cause planes to fall out of the sky.
After trying to sort out this mess and giving Certipost time to transition,
we've decided to revoke the issuing certificates the last week of November.
We believe there are over 400 outstanding certificates that will be


5.     Postecom refused an audit and decided to exit operation of a CA. We
have not yet revoked their issuing CAs in order to give them time to migrate
to a different CA infrastructure. They asked that we keep the issuing CAs
active until December 31, 2016 to complete the migration. Although they have
worked on migrating certs, Postecom still has 1185 certificates to migrate
in the next two months. The issuing CAs are scheduled for revocation on
January 5th.  


6.     Vodafone completed a Webtrust audit this year. Unfortunately, we
found out that the audit has qualifications.  The audit report hasn't issued
yet but should later in November. Once the audit report issues, we will
provide a copy of the report for evaluation. We've been told that the
qualifications are related to the network security guidelines. They probably
also include the SHA1 issuance, although we haven't been told that directly.
Vodafone is currently designing and building a new CA infrastructure that
will house all of their operations. To ensure compliance, we are requiring
the new infrastructure to undergo a point-in-time audit in December. We are
also requiring a full audit on their issuance processes at the start of next
year. A full, non-qualified audit report is expected by March 31, 2016.
Although this is well past the compliance date, we are sympathetic to
Vodafone as they have been actively working towards compliance. They also
have 4,580 public facing SSL certs makes revocation tricky. They issue a lot
of client certificates from these sub CAs where revocation could have a
potentially devastating effect on telecommunications. Because of the large
impact of revocation and how hard Vodafone is working to migrate, we'd like
permission for them to comply by March 31, 2017, assuming serious issues are
not uncovered by the audit report and that the December and January audits
pass unqualified.  


7.       The Belgium government is our biggest challenge in migrating
Verizon customers. With over 20 issuing CAs, Belgium has the largest
outstanding non-compliant infrastructure. The operators have also claimed
that revoking their issuing CAs is illegal (in Belgium). The government is
using the issuing CA for creating personal identification (e-ID) cards
throughout the country. The Belgium government has dictated that they set
the rules, not us. Although the Belgium government does not have an audit
yet, Verizon has represented that the issuing CAs are hosted in the Verizon
infrastructure and are potentially covered by the Verizon audit. We've asked
Verizon to provide an updated audit report showing coverage of the Belgium
issuing CAs by December 1, 2016. If the report is not delivered by December
1, 2016, we plan to immediately revoke the issuing CAs.  If, for whatever
reason, we are unable to revoke the issuing CAs at that time, we would
certainly not object to the browsers distrusting the issuing CAs issued to


Resolving these remaining seven issues will, bring all Verizon's sub CAs
into compliance with the audit requirements by March 31, 2017. As of January
2017, all sub CAs operated under the Verizon umbrella will have a
non-qualified audit with the exception of Vodafone, which will have a
non-qualified point-in-time audit by then. 



Although by March 31, 2017, all former Verizon customers will comply with
the audit portion of the BRs, experience has recently shown that strict
compliance of profile requirements is more difficult to enforce. Such is the
case of the SHA1 certificates issued by Nets Norway, Siemens, and Vodafone.
Nets Norway and Siemens were both ETSI audited while Vodafone opted for a
Webtrust audit.   


Shortly after the Verizon transaction, we sent a statement to each CA
announcing the acquisition and our expectation of strict compliance with the
BRs and network security requirements. On January 27, 2016, we sent another
email to all CAs that reiterated the need for strict adherence to standards
and requirement for CAs to update their infrastructure as needed to maintain
an unqualified audit. The specifically stated that issuance of internal name
certificates and SHA1 certificates is strictly forbidden. Nets Norway,
Vodafone, and Siemens all received a copy of this email.  


As all the sub CAs should have been aware of the requirements in 2015, were
reminded this year, and had audits of their infrastructure, we are very
disappointed by the recently discovered SHA-1 issuance. We've contacted each
of the sub CAs asking for an explanation on what happened and why their
infrastructure wasn't sufficient to prevent SHA1 issuance.  


Of the SHA1 cert issuers, Vodafone is the only sub CA that responded by
providing a timeline for compliance, information about how they are
migrating to new CA (which we already knew), and a plan going forward to
undergo an audit and make sure technical constraints are active to prevent
additional SHA1 certificates. Siemens replied that they are revoking the
SHA1 certificates, that issuance was a mistake, and that they are taking
action to prevent additional mis-issuance. However, the specifics of their
plan are vague (other than providing operational training to prevent
reoccurrence). For both Vodafone and Siemens, this is their first offense in
mis-issuing a SHA1 certificate, and we'd ask for leniency. We do recognize
that this is a serious concern and can revoke the issuing CAs if the action
is decided necessary. For full disclosure, Siemens is transitioning away
from the Verizon root certificate to another CA. We are only maintained the
validity of their cross-sign in a good faith effort to help them complete
the transition. We were planning to revoke the issuing CAs early next year
(January 5th) once the transition is completed.  


The final CA is Nets Norway. Unlike the other two issuing CAs, Nets Norway
issued a SHA1 certificate earlier this year. Although Nets Norway promptly
revoked the certificate and provided verbal assurances that SHA1 issuance
would be restricted, the company obviously failed to take adequate measures.
We informed Nets Norway that we are revoking their issuing CA later this
month. We haven't revoked them yet because we wanted to complete the
incident report first and allow public comment before finalizing the


Our investigation of Verizon's, Postecom's, and Telecom Italia's issuance of
a SHA1 certificate is ongoing. As mentioned above, Postecom is scheduled for
revocation. Our plan for now is to keep the current revocation schedule for
Postecom as the SHA1 issuance occurred twice during January.  We haven't
formed a plan for Verizon or Telecom Italia yet and are waiting to hear back
from their representatives. Verizon issued a single certificate back in
January. Telecom Italia issued three in early January and February. Each of
these incidents were only recently discovered by DigCert. We've since added
an active monitor to crt.sh to alert our compliance team if a SHA1
certificate issues.



Hopefully this helps explain the status of the Verizon transition and the
recent SHA1 incidents. We are still collecting information from the three
issuing sub CAs about what happened, but I hope this is enough to provide
transparency and make a decision on how to proceed.  


Despite the difficulties in bringing these customers into compliance, we
still think the Verizon transaction was a good move from a compliance
perspective. Helping these sub CAs identify problems and improve their
processes has been an enlightening experience and given us insight on some
of the difficulties other CAs may face in complying with the BRs. The
process has been interesting exercise in helping others make remediation
plans and trying to accelerate what feels like a slow-moving project.
Despite the long nights, headaches, and frustrations, I think we've been
successful in our original goal of improving the security ecosystem. I'm
also very pleased that the end of the transition is almost here!  


We appreciate your feedback and look forward to your comments.  









Attachment: smime.p7s
Description: S/MIME cryptographic signature

dev-security-policy mailing list

Reply via email to