Re: How do you handle mass revocation requests?

[Ryan Sleevi via dev-security-policy]

On Thu, Mar 1, 2018 at 10:31 AM, Nick Lamb via dev-security-policy <
dev-security-policy@lists.mozilla.org> wrote:

> On Thu, 1 Mar 2018 10:51:04 +
> Ben Laurie via dev-security-policy
>  wrote:
>
> > Seems to me that signing something that has nothing to do with certs
> > is a safer option - e.g. sign random string+Subject DN.
>
> That does sounds sane, I confess I have not spent much time playing with
> easily available tools to check what is or is not easily possible on
> each platform in terms of producing and checking such proofs. I knew
> that you can make a CSR on popular platforms, and I knew how to check a
> CSR is valid and a bogus CSR seemed obviously harmless to me.
>
> I feel sure I saw someone's carefully thought through procedure for
> proving control over a private key written up properly for close to
> this sort of situation but I have tried and failed to find it again
> since the incident was first reported, and apparently Jeremy didn't
> know it either.


Perhaps you were thinking about the ROBOT attack, which covered
https://robotattack.org/

You can produce such messages, if you have the key, through something like
(note: haven't tested, but you get the idea)

associated_spki_hash=`openssl pkey -inform PEM -in foo.key -pubout |
openssl dgst -sha256 -binary | openssl enc -base64`
associated_crt_sh_url="http://crt.sh/?spkisha256=${associated_spki_hash};
echo "${associated_crt_sh_url} is compromised - 2018-03-01" > message.txt
openssl dgst -sha256 -sign foo.key -out foo.key -out
${associated_spki_hash}.signature message.txt

to verify
openssl dgst -sha256 -verify <(openssl x509 -in "cert file goes here"
-pubkey -noout) -signature ${associated_spki_hash}.signature message.txt
___
dev-security-policy mailing list
dev-security-policy@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-security-policy

Re: How do you handle mass revocation requests?

[Ryan Duff via dev-security-policy]

On Thursday, March 1, 2018 at 11:08:58 AM UTC-5, RSTS wrote:
> On Thursday, March 1, 2018 at 1:51:16 PM UTC, Michel Gre wrote:
> > > I'd postulate there's
> > > nothing wrong with Trustico holding the private keys if they were hosting
> > > the site or providing CDN services for all of these sites. 
> > 
> > I manage one of the affected domains. I can tell that in no way does 
> > Trustico hosts the site, nor provide us any CDN service.
> > 
> > We just purchased them a certificate 4 years ago and renewed it for 3 years 
> > in april 2015. Since we are usually quite busy we simply used their form to 
> > generate the key, the CSR, and get the certificate... So, Trustico should 
> > be actually Dontrustico. The worst is that the CEO himself publicly said 
> > (here!) that they HELD OUR PRIVATE KEYS!!! Come on. M. Zane Lucas, your 
> > staff sent me (after I asked them from an explanation regarding the 
> > Digicert's first email) a coupon for a "Trustico(r) Single Site" 
> > certificate, would you expect me to trust it after what YOU disclosed here? 
> > Looks like you just cut the branch your company was sitting on.
> 
> In relevant news, Trustico's site is down due to an apparent flaw, apparently 
> allowing users to run commands as root on their production webserver. 
> 
> My question is, assuming this was discovered previously by an attacker, is 
> there possibility of exploiting that to fetch these cold-storage keys?
> 
> https://twitter.com/Manawyrm/status/969230542578348033 in reply to 
> https://twitter.com/svblxyz/status/969220402768736258

Given that they were able to readily produce all of these keys, I would suspect 
they were never really in cold storage. At least not exclusively.
___
dev-security-policy mailing list
dev-security-policy@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-security-policy

Re: How do you handle mass revocation requests?

[RSTS via dev-security-policy]

On Thursday, March 1, 2018 at 1:51:16 PM UTC, Michel Gre wrote:
> > I'd postulate there's
> > nothing wrong with Trustico holding the private keys if they were hosting
> > the site or providing CDN services for all of these sites. 
> 
> I manage one of the affected domains. I can tell that in no way does Trustico 
> hosts the site, nor provide us any CDN service.
> 
> We just purchased them a certificate 4 years ago and renewed it for 3 years 
> in april 2015. Since we are usually quite busy we simply used their form to 
> generate the key, the CSR, and get the certificate... So, Trustico should be 
> actually Dontrustico. The worst is that the CEO himself publicly said (here!) 
> that they HELD OUR PRIVATE KEYS!!! Come on. M. Zane Lucas, your staff sent me 
> (after I asked them from an explanation regarding the Digicert's first email) 
> a coupon for a "Trustico(r) Single Site" certificate, would you expect me to 
> trust it after what YOU disclosed here? Looks like you just cut the branch 
> your company was sitting on.

In relevant news, Trustico's site is down due to an apparent flaw, apparently 
allowing users to run commands as root on their production webserver. 

My question is, assuming this was discovered previously by an attacker, is 
there possibility of exploiting that to fetch these cold-storage keys?

https://twitter.com/Manawyrm/status/969230542578348033 in reply to 
https://twitter.com/svblxyz/status/969220402768736258
___
dev-security-policy mailing list
dev-security-policy@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-security-policy

Re: How do you handle mass revocation requests?

[Nick Lamb via dev-security-policy]

On Thu, 1 Mar 2018 10:51:04 +
Ben Laurie via dev-security-policy
 wrote:

> Seems to me that signing something that has nothing to do with certs
> is a safer option - e.g. sign random string+Subject DN.

That does sounds sane, I confess I have not spent much time playing with
easily available tools to check what is or is not easily possible on
each platform in terms of producing and checking such proofs. I knew
that you can make a CSR on popular platforms, and I knew how to check a
CSR is valid and a bogus CSR seemed obviously harmless to me.

I feel sure I saw someone's carefully thought through procedure for
proving control over a private key written up properly for close to
this sort of situation but I have tried and failed to find it again
since the incident was first reported, and apparently Jeremy didn't
know it either.
___
dev-security-policy mailing list
dev-security-policy@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-security-policy

Re: How do you handle mass revocation requests?

[nic.swart--- via dev-security-policy]

I agree with Eric, I would call storing the customers private keys (without 
their knowledge!!) as an immediate compromise and a clear breach of trust.

On Thursday, March 1, 2018 at 1:04:54 AM UTC+1, Eric Mill wrote:
> Trustico doesn't seem to provide any hosting or CDN services that would
> make use of the private key, nor do they appear to explicitly inform users
> about the storage of this private key.
> 
> In their statement, they say they keep the private keys explicitly to
> perform revocation as necessary:
> https://www.trustico.com/news/2018/symantec-revocation/certificate-replacement.php
> (archived: https://archive.is/0AnyR )
> 
> > These Private Keys are stored in cold storage, for the purpose of
> revocation.
> 
> Their CSR/key generation form is here:
> https://www.trustico.com/ssltools/create/csr-pem/create-a-new-csr-instantly.php
> (archived: https://archive.is/hJV42 )
> 
> The storage of private keys appears to be done without the user's knowledge
> or consent. And of course, only the keys they create through their form are
> stored, so it is clearly not a necessary business function for most of
> their certificate business.
> 
> Finally -- the CSR/key generation form page incorporates JavaScript from at
> least 5 or 6 different companies (including ad servers), which would allow
> any of those third parties (intentionally or through compromise of their
> own) to capture generated keys. This is a reckless amount of exposure, to
> the point that even if the keys were generated completely inside the
> browser and never exposed to the server (which does not appear to be the
> case), I would consider them compromised at the time of generation.
> 
> Given everything that's known, then regardless of who emailed whose
> customers when and why, I think it's clear that Trustico compromised those
> keys at _least_ at the time they were stored, if not at the time of
> generation, and has been routinely compromising customer keys for years.
> Emailing them to DigiCert only widened their exposure to more unauthorized
> parties.
> 
> And given that there's no evidence that Trustico has acknowledged this
> fact, or indicated any intent to change their business practices, then I
> believe it's appropriate for all CAs to immediately suspend or terminate
> their relationship with Trustico -- as any CA who continued doing business
> with Trustico would now be knowingly allowing Trustico to compromise the
> keys of the certificates issued under their hierarchy.
> 
> -- Eric
___
dev-security-policy mailing list
dev-security-policy@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-security-policy

RE: How do you handle mass revocation requests?

[Michel Gre via dev-security-policy]

> I'd postulate there's
> nothing wrong with Trustico holding the private keys if they were hosting
> the site or providing CDN services for all of these sites. 

I manage one of the affected domains. I can tell that in no way does Trustico 
hosts the site, nor provide us any CDN service.

We just purchased them a certificate 4 years ago and renewed it for 3 years in 
april 2015. Since we are usually quite busy we simply used their form to 
generate the key, the CSR, and get the certificate... So, Trustico should be 
actually Dontrustico. The worst is that the CEO himself publicly said (here!) 
that they HELD OUR PRIVATE KEYS!!! Come on. M. Zane Lucas, your staff sent me 
(after I asked them from an explanation regarding the Digicert's first email) a 
coupon for a "Trustico(r) Single Site" certificate, would you expect me to 
trust it after what YOU disclosed here? Looks like you just cut the branch your 
company was sitting on.
___
dev-security-policy mailing list
dev-security-policy@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-security-policy

Re: How do you handle mass revocation requests?

[Rob Stradling via dev-security-policy]

On 01/03/18 10:51, Ben Laurie via dev-security-policy wrote:

On 28 February 2018 at 21:37, Nick Lamb via dev-security-policy <
dev-security-policy@lists.mozilla.org> wrote:


On Wed, 28 Feb 2018 20:03:51 +
Jeremy Rowley via dev-security-policy
 wrote:


The keys were emailed to me. I'm trying to get a project together
where we self-sign a cert with each of the keys and publish them.
That way there's evidence to the community of the compromise without
simply listing 23k private keys. Someone on Reddit suggested that,
which I really appreciated.


That's probably me (tialaramex).

Anyway, if it is me you're referring to, I suggested using the private
keys to issue a bogus CSR. CSRs are signed, proving that whoever made
them had the corresponding private key but they avoid the confusion
that comes from DigiCert (or its employees) issuing bogus certs.
Everybody reading m.d.s.policy can still see that a self-signed cert is
harmless and not an attack, but it may be harder to explain in a
soundbite. Maybe more technically able contributors disagree ?



Seems to me that signing something that has nothing to do with certs is a
safer option - e.g. sign random string+Subject DN.


And also throw in some transparency...

https://mailarchive.ietf.org/arch/msg/trans/WLFmIyaH4BJo77ZJDinKJcylOcg

--
Rob Stradling
Senior Research & Development Scientist
Email: rob.stradl...@comodoca.com
Bradford, UK
Office: +441274730505
ComodoCA.com
___
dev-security-policy mailing list
dev-security-policy@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-security-policy

Re: How do you handle mass revocation requests?

[Ben Laurie via dev-security-policy]

On 28 February 2018 at 21:37, Nick Lamb via dev-security-policy <
dev-security-policy@lists.mozilla.org> wrote:

> On Wed, 28 Feb 2018 20:03:51 +
> Jeremy Rowley via dev-security-policy
>  wrote:
>
> > The keys were emailed to me. I'm trying to get a project together
> > where we self-sign a cert with each of the keys and publish them.
> > That way there's evidence to the community of the compromise without
> > simply listing 23k private keys. Someone on Reddit suggested that,
> > which I really appreciated.
>
> That's probably me (tialaramex).
>
> Anyway, if it is me you're referring to, I suggested using the private
> keys to issue a bogus CSR. CSRs are signed, proving that whoever made
> them had the corresponding private key but they avoid the confusion
> that comes from DigiCert (or its employees) issuing bogus certs.
> Everybody reading m.d.s.policy can still see that a self-signed cert is
> harmless and not an attack, but it may be harder to explain in a
> soundbite. Maybe more technically able contributors disagree ?
>

Seems to me that signing something that has nothing to do with certs is a
safer option - e.g. sign random string+Subject DN.
___
dev-security-policy mailing list
dev-security-policy@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-security-policy

Re: How do you handle mass revocation requests?

[Ben Laurie via dev-security-policy]

On 28 February 2018 at 19:40, Jeremy Rowley via dev-security-policy <
dev-security-policy@lists.mozilla.org> wrote:

> The end user agreed to the subscriber agreement, not Trustico. Our
> analysis follows what Peter B. posted – the subscriber is the “natural
> person or Legal Entity to whom a Certificate is issued and who is legally
> bound by a Subscriber Agreement or Terms of Use"—which in this case was
> Trustico’s customers.  In addition, we felt that given (1) the number of
> certificates Trustico was asking us to mass-revoke and (2) the lack of any
> substantiation of why Trustico thought the certs were “compromised,” we
> needed more information before revoking. At the minimum, it warranted
> alerting the contact for each certificate that revocation was imminent.
>
>
>
> I also agree that there’s no problem with the way or that the keys were
> provided to DigiCert for cert revocation.


Agree with who? Both asking for the keys and providing them seems weird to
me.

The more secure thing to do would be to ask for proof of possession of the
keys, e.g. by signing a random string with them...


> I certainly don’t want to discourage revocation of compromised certs!  My
> main question is how do you handle these things? The process at scale
> should not be different if a reseller requests revocation compared to any
> other security researcher. The question is how you handle the this volume
> of revocations when its happen? I’ve never received a revocation request of
> this magnitude before so I’m seeking the wisdom of the community in what we
> do.
>
>
>
> I’m happy to share any of the details I can.
>
>
>
> Jeremy
>
>
>
>
>
> From: Ryan Sleevi <r...@sleevi.com>
> Sent: Wednesday, February 28, 2018 11:58 AM
> To: Jeremy Rowley <jeremy.row...@digicert.com>
> Cc: mozilla-dev-security-pol...@lists.mozilla.org
> Subject: Re: How do you handle mass revocation requests?
>
>
>
>
>
>
>
> On Wed, Feb 28, 2018 at 12:37 PM, Jeremy Rowley via dev-security-policy <
> dev-security-policy@lists.mozilla.org <mailto:dev-security-policy@
> lists.mozilla.org> > wrote:
>
> On February 2nd, 2018, we received a request from Trustico to mass revoke
> all certificates that had been ordered by end users through Trustico.
> Unfortunately, the email was not sent to the appropriate certificate
> problem
> reporting channels and did not surface immediately so we're delayed in
> sharing the concerns and information. Once we were alerted, the team kicked
> off a debate that I wanted to bring to the CAB Forum. Basically, our
> position is that resellers do not constitute subscribers under the Baseline
> Requirement's definitions (Section 1.6.1). As such, we needed to confirm
> that either the key was compromised or that they revocation was authorized
> by the domain holder (the subscriber) prior to revoking the certificate.
> The
> certificates were not alleged as compromised at that time.
>
>
>
> I think there's a little nuance to this in the general case (e.g. consider
> "Resllers" who are also the hosting provider, and thus constitute the
> Applicant/Subscriber even when they're not the domain holder, but
> authorized by them), but based on this specific case, I think this sounds
> like a correct determination.
>
>
>
> I think the biggest question is who agreed to the terms - Trustico or the
> end-user - and that mostly impacts the rest of the decision here. Further,
> did Trustico warrant on behalf of the user that the user agreed to the
> terms (in which case, I would think it's a bit of a copout, and it's really
> Trustico agreeing, thus the Subscriber), or did DigiCert have direct
> communication with the user on the basis of Trustico's introduction (in
> which case, yeah, the Subscriber is the user)
>
>
>
> Anyway, just highlighting it here as perhaps not a uniform consensus, but
> perhaps not as material as what follows.
>
>
>
> On 2/27/2018, at my request for proof of compromise, we received a file
> with
> 23k private keys matched to specific Trustico customers. This definitely
> triggered our 24-hour revocation processing requirement under 4.9.1.1.3.
> Once we received the keys, we confirmed that these were indeed the matching
> private keys for the reported certificates. We will be revoking these
> certificates today (February 28th, 2018).
>
>
>
> I think all of this sounds reasonable, no different than a security
> researcher (or random member of the public) who were to claim compromise
> with no evidence of that.
>
>
>
> Currently, we are only revoking the certificates if we received the private
> keys. There are additional certificates the reseller request

Re: How do you handle mass revocation requests?

[Eric Mill via dev-security-policy]

Trustico doesn't seem to provide any hosting or CDN services that would
make use of the private key, nor do they appear to explicitly inform users
about the storage of this private key.

In their statement, they say they keep the private keys explicitly to
perform revocation as necessary:
https://www.trustico.com/news/2018/symantec-revocation/certificate-replacement.php
(archived: https://archive.is/0AnyR )

> These Private Keys are stored in cold storage, for the purpose of
revocation.

Their CSR/key generation form is here:
https://www.trustico.com/ssltools/create/csr-pem/create-a-new-csr-instantly.php
(archived: https://archive.is/hJV42 )

The storage of private keys appears to be done without the user's knowledge
or consent. And of course, only the keys they create through their form are
stored, so it is clearly not a necessary business function for most of
their certificate business.

Finally -- the CSR/key generation form page incorporates JavaScript from at
least 5 or 6 different companies (including ad servers), which would allow
any of those third parties (intentionally or through compromise of their
own) to capture generated keys. This is a reckless amount of exposure, to
the point that even if the keys were generated completely inside the
browser and never exposed to the server (which does not appear to be the
case), I would consider them compromised at the time of generation.

Given everything that's known, then regardless of who emailed whose
customers when and why, I think it's clear that Trustico compromised those
keys at _least_ at the time they were stored, if not at the time of
generation, and has been routinely compromising customer keys for years.
Emailing them to DigiCert only widened their exposure to more unauthorized
parties.

And given that there's no evidence that Trustico has acknowledged this
fact, or indicated any intent to change their business practices, then I
believe it's appropriate for all CAs to immediately suspend or terminate
their relationship with Trustico -- as any CA who continued doing business
with Trustico would now be knowingly allowing Trustico to compromise the
keys of the certificates issued under their hierarchy.

-- Eric

On Wed, Feb 28, 2018 at 3:24 PM, Ryan Hurst via dev-security-policy <
dev-security-policy@lists.mozilla.org> wrote:

> On Wednesday, February 28, 2018 at 11:56:04 AM UTC-8, Ryan Sleevi wrote:
> > Assuming Trustico sent the keys to DigiCert, it definitely sounds like
> even
> > if Trustico was authorized to hold the keys (which is a troubling
> argument,
> > given all things), they themselves compromised the keys of their
> customers,
> > and revocation is both correct and necessary. That is, whether or not
> > Trustico believed they were compromised before, they compromised their
> > customers keys by sending them, and it's both correct and accurate to
> > notify the Subscribers that their keys have been compromised by their
> > Reseller.
>
> That seems to be the case to me as well.
>
> It also seems that this situation should result in the UAs and/or CABFORUM
> re0visit section 6.1.2 (https://github.com/cabforum/
> documents/blob/master/docs/BR.md) in the BRs.
>
> Specifically, this section states:
>
> ```
> Parties other than the Subscriber SHALL NOT archive the Subscriber Private
> Key without authorization by the Subscriber.
>
> If the CA or any of its designated RAs generated the Private Key on behalf
> of the Subscriber, then the CA SHALL encrypt the Private Key for transport
> to the Subscriber.
> ```
>
> In this case, TrustIco is not the subscriber, and there is no indication
> in their terms and conditions (https://www.trustico.com/
> terms/terms-and-conditions.php) that they are authorized to archive the
> private key. Yet clearly if they were able to provide 20k+ private keys to
> DigiCert they are archiving them. This text seems to cover this case
> clearly but as worded I do not see how audits would catch this behavior. I
> think it may make sense for the CAs to be responsible for demonstrating how
> they and other non-subscribers in the lifecycle flow handle this case.
>
> Additionally, it seems if the private keys were provided to DigiCert in a
> way they were verifiable by them they may have been stored in a
> non-encrypted fashion, at a minimum they were likley not generated and
> protected on an HSM. The BRs should probably be revised to specify some
> minimum level of security to be provided in these cases of for these cases
> to be simply disallowed altogether.
>
> Finally, the associated text speaks to RAs but not to the non-subscriber
> (reseller) case, this gap should be addressed minimally.
> ___
> dev-security-policy mailing list
> dev-security-policy@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-security-policy
>



-- 
konklone.com | @konklone 
___
dev-security-policy mailing list

Re: How do you handle mass revocation requests?

[Matthew Hardeman via dev-security-policy]

On Wednesday, February 28, 2018 at 4:44:50 PM UTC-6, Jeremy Rowley wrote:
> 1) Not all of the certificates being revoked use the Symantec hierarchy.
> There are some certs that use the DigiCert replacement hierarchy. Not many
> though.
> 2) Sorry my wording was strange.  It almost always is.  What I meant, is
> Trustico specifically asked for the certs to be revoked within 24 hours as
> required by the BRs. They said in the email they were triggering the 24 hour
> requirement. 
> 3) I really feel like a third point would be important, but I can't think of
> one. 

So, I suspect what I and others in the community, like Mr. Duff, are wondering 
is: Why did Trustico go to such lengths to forcibly cause revocation of all of 
these certificates?

Because their initial request was for the revocation to occur and Digicert's 
initial position was that they are not the subscriber and so didn't have to 
revoke at their request they finally had someone put together the keys and 
re-request with the formal 24 hour deadline.  That sounds like they went to 
significant effort to ensure the revocation would happen.

If the keys were truly leaked or otherwise compromised, it makes sense.  If 
they were not and Trustico just pulled them out to trigger the 24 hour 
revocation window, why did they do that?

It seems like the only people they're (directly) hurting are their own 
customers.  That's quite irrational behavior and generally businesses do not so 
overtly act against their own interest.

That sort of thing will scare a customer away from a reseller in a hurry.

So the question on my mind is, why would Trustico do that to their own 
customers?  It can't just be about migration to a new CA, right?  If so, 
wouldn't the certificates just be left valid and new certificates be issued by 
the new CA?

Forcing the 24 hour revocation window just creates a support nightmare for 
Trustico and Trustico's customers with no apparent upside, unless there was a 
real compromise.

Trustico's customers would seem to be entitled to know whether Trustico 
actually suffered a breach which compromises those customers.
___
dev-security-policy mailing list
dev-security-policy@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-security-policy

RE: How do you handle mass revocation requests?

[Jeremy Rowley via dev-security-policy]

1) Not all of the certificates being revoked use the Symantec hierarchy.
There are some certs that use the DigiCert replacement hierarchy. Not many
though.
2) Sorry my wording was strange.  It almost always is.  What I meant, is
Trustico specifically asked for the certs to be revoked within 24 hours as
required by the BRs. They said in the email they were triggering the 24 hour
requirement. 
3) I really feel like a third point would be important, but I can't think of
one. 

-Original Message-
From: dev-security-policy
<dev-security-policy-bounces+jeremy.rowley=digicert@lists.mozilla.org>
On Behalf Of Matthew Hardeman via dev-security-policy
Sent: Wednesday, February 28, 2018 3:23 PM
To: mozilla-dev-security-pol...@lists.mozilla.org
Subject: Re: How do you handle mass revocation requests?

On Wednesday, February 28, 2018 at 3:55:37 PM UTC-6, Ryan Duff wrote:
> >From what I've read, it appears the situation here is that Trustico
wanted to revoke all their customer certs from Digicert so they could do a
mass migration to another CA (which is not a proper reason to revoke). When
asked for proof by Digicert that the certificates were compromised and
needed to be revoked, Trustico sent Digicert 23,000(!) private keys that
*they had stored* due to the fact that they were generated by their
web-based system in order to effectively *make them* compromised.
> 
> Am I missing anything?

That's kind of what I was thinking happened also.

Though a couple points to correct:  The original issuing CA hierarchy is a
Symantec trust path.  This suggests that what they really wanted to occur
was to trigger a 24 hour reissue of all of these certificates under a
DigiCert trusted path -- since presumably any issuance at this point would
fall under a DigiCert path.

Thus, within 24 hours, getting new certificates for all their customers
under the new trust path.  I'm going to guess someone at Trustico was
getting annoyed at support calls regarding the migration and somehow assumed
there'd be no consequences for pushing the issue by way of getting all those
certificates revoked on "security" grounds.

As grounds for this belief, I submit the strangely worded statement of Mr.
Rowley at the start of the thread "Later, the company shared with us that
they held the private keys and the certificates were compromised, trying to
trigger the BR's 24-hour revocation requirement".

That language seems to imply that there's a sense that the security / web
PKI integrity aspect is less the matter at stake and more that the keys were
located and sent over to create an impossible to ignore security issue
forcing the 24 hour window.

My guess is that the person at Trustico wanted immediate reissuance of all
of the Symantec certificates under the DigiCert trust paths and assumed:

1.  That revoking the certs for security reasons would result in ASAP
reissue (probably true in one-offs).
2.  That the reissuance would happen in the DigiCert trust path (almost
certainly true).
3.  That they'd have a spike of support issues related to the reissuances,
but that Trustico would have more control over the period over which they
had to help customers migrate certificates and then the "bleeding" would
stop.
___
dev-security-policy mailing list
dev-security-policy@lists.mozilla.org
https://clicktime.symantec.com/a/1/1SicdYM5NOSV8S29ZlemtDgUftWWe-mEnCeFfDCfA
lw=?d=paMnJZlER7IHb626K-31V2u-Kf4DBs2hoO7Bro78ZuywH3EZ9N1dve7JXTCFPZFHyQrvw7
cDEqjPm1CCO0iQqqCbe-J7q2_uVOXTQzSslJpmeaUe9RmpgB81xaobKJOyb_YpAY8IOdkE832w7N
tu7J94BmAMUFyIN-LINYJhcdKrmRggiP3dwENTjoH8GFBgeJgbAAJ7AXEY8EsaOLt-dqVymUiml9
GCqK8Pz2bZYq21i2TjrlH5i5ctTnHGcaJLrBukwohzJ1XA0B0Ma6sfh6NdDO9yi3psZCXUIDcZyp
5FtLBoYtTo6DUQVC8VhfOJY6KGhSpursgUnQdy6yQRtirjLO9kmIOKJSuJvlbaKD3gYKkxIRzMLA
6jPbVCxU77Z6q6OAnzLtAIRiCe7-MVzty2jFdcG8R9uiVI7Vf241-3ANcBFbr288JrDaZYHwhYJu
rWo3wUC3HNJXFV_nf74MJE=https%3A%2F%2Flists.mozilla.org%2Flistinfo%2Fdev-se
curity-policy


smime.p7s
Description: S/MIME cryptographic signature
___
dev-security-policy mailing list
dev-security-policy@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-security-policy

Re: How do you handle mass revocation requests?

[Ryan Sleevi via dev-security-policy]

On Wed, Feb 28, 2018 at 5:23 PM, Matthew Hardeman via dev-security-policy <
dev-security-policy@lists.mozilla.org> wrote:

> On Wednesday, February 28, 2018 at 3:55:37 PM UTC-6, Ryan Duff wrote:
> > >From what I've read, it appears the situation here is that Trustico
> wanted to revoke all their customer certs from Digicert so they could do a
> mass migration to another CA (which is not a proper reason to revoke). When
> asked for proof by Digicert that the certificates were compromised and
> needed to be revoked, Trustico sent Digicert 23,000(!) private keys that
> *they had stored* due to the fact that they were generated by their
> web-based system in order to effectively *make them* compromised.
> >
> > Am I missing anything?
>
> That's kind of what I was thinking happened also.
>
> Though a couple points to correct:  The original issuing CA hierarchy is a
> Symantec trust path.  This suggests that what they really wanted to occur
> was to trigger a 24 hour reissue of all of these certificates under a
> DigiCert trusted path -- since presumably any issuance at this point would
> fall under a DigiCert path.
>
> Thus, within 24 hours, getting new certificates for all their customers
> under the new trust path.  I'm going to guess someone at Trustico was
> getting annoyed at support calls regarding the migration and somehow
> assumed there'd be no consequences for pushing the issue by way of getting
> all those certificates revoked on "security" grounds.
>
> As grounds for this belief, I submit the strangely worded statement of Mr.
> Rowley at the start of the thread "Later, the company shared with us that
> they held the private keys and the
> certificates were compromised, trying to trigger the BR's 24-hour
> revocation
> requirement".
>
> That language seems to imply that there's a sense that the security / web
> PKI integrity aspect is less the matter at stake and more that the keys
> were located and sent over to create an impossible to ignore security issue
> forcing the 24 hour window.
>
> My guess is that the person at Trustico wanted immediate reissuance of all
> of the Symantec certificates under the DigiCert trust paths and assumed:
>
> 1.  That revoking the certs for security reasons would result in ASAP
> reissue (probably true in one-offs).
> 2.  That the reissuance would happen in the DigiCert trust path (almost
> certainly true).
> 3.  That they'd have a spike of support issues related to the reissuances,
> but that Trustico would have more control over the period over which they
> had to help customers migrate certificates and then the "bleeding" would
> stop.


Note: The evidence on this thread does not support those assumptions. Among
other things, Trustico no longer is a reseller of Symantec (now DigiCert)
certificates, and has transitioned new issuance to Comodo. This was already
covered in
https://groups.google.com/d/msg/mozilla.dev.security.policy/wxX4Yv0E3Mk/jx6r9jlPAwAJ
___
dev-security-policy mailing list
dev-security-policy@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-security-policy

Re: How do you handle mass revocation requests?

[Matthew Hardeman via dev-security-policy]

On Wednesday, February 28, 2018 at 3:55:37 PM UTC-6, Ryan Duff wrote:
> >From what I've read, it appears the situation here is that Trustico wanted 
> >to revoke all their customer certs from Digicert so they could do a mass 
> >migration to another CA (which is not a proper reason to revoke). When asked 
> >for proof by Digicert that the certificates were compromised and needed to 
> >be revoked, Trustico sent Digicert 23,000(!) private keys that *they had 
> >stored* due to the fact that they were generated by their web-based system 
> >in order to effectively *make them* compromised.
> 
> Am I missing anything?

That's kind of what I was thinking happened also.

Though a couple points to correct:  The original issuing CA hierarchy is a 
Symantec trust path.  This suggests that what they really wanted to occur was 
to trigger a 24 hour reissue of all of these certificates under a DigiCert 
trusted path -- since presumably any issuance at this point would fall under a 
DigiCert path.

Thus, within 24 hours, getting new certificates for all their customers under 
the new trust path.  I'm going to guess someone at Trustico was getting annoyed 
at support calls regarding the migration and somehow assumed there'd be no 
consequences for pushing the issue by way of getting all those certificates 
revoked on "security" grounds.

As grounds for this belief, I submit the strangely worded statement of Mr. 
Rowley at the start of the thread "Later, the company shared with us that they 
held the private keys and the 
certificates were compromised, trying to trigger the BR's 24-hour revocation 
requirement".

That language seems to imply that there's a sense that the security / web PKI 
integrity aspect is less the matter at stake and more that the keys were 
located and sent over to create an impossible to ignore security issue forcing 
the 24 hour window.

My guess is that the person at Trustico wanted immediate reissuance of all of 
the Symantec certificates under the DigiCert trust paths and assumed:

1.  That revoking the certs for security reasons would result in ASAP reissue 
(probably true in one-offs).
2.  That the reissuance would happen in the DigiCert trust path (almost 
certainly true).
3.  That they'd have a spike of support issues related to the reissuances, but 
that Trustico would have more control over the period over which they had to 
help customers migrate certificates and then the "bleeding" would stop.
___
dev-security-policy mailing list
dev-security-policy@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-security-policy

Re: How do you handle mass revocation requests?

[Jeremy Rowley via dev-security-policy]

No. Just the 23k. Part of the reason I posted to the Mozilla list are open 
questions about whether Trusticos request is sufficient to trigger the br 
requirements. My gut is no, and sounds like the browsers agree.  We’ll only 
revoke the remaining 27k if we receive evidence of compromise

On Feb 28, 2018, at 2:52 PM, jomo 
<mozilla-li...@jomo.tv<mailto:mozilla-li...@jomo.tv>> wrote:


Basically, we're revoking 50k people without
being able to explain why (well, other than the key was compromised).

Unless I misunderstood, you originally said you received 23k compromised keys 
and are revoking those.

Currently, we are only revoking the certificates if we received the private
keys. There are additional certificates the reseller requested to have
revoked, but DigiCert has decided to disregard that request until we receive
proof of compromise or more information about the cause of this incident.


Has DigiCert received proof of compromise of all 50k in the meantime?


On 28.2.18 22:42, Jeremy Rowley via dev-security-policy wrote:

We don't have a process to prevent third parties from storing private keys.
I'm not sure how that would even work considering the approved third-party
use cases vs. non-approved use cases.  In fact, I'd postulate there's
nothing wrong with Trustico holding the private keys if they were hosting
the site or providing CDN services for all of these sites. The issue is
Trustico alleged compromise of the certificates and sent us the private keys
believed compromised in support. There were a lot of them.

This is a mass revocation without any explanation of what went wrong or why.
The reseller mentioned and proved compromise, but there's no way to see if
what happened, whether the issue was mitigated, or how it's going to be
prevented from happening again. Basically, we're revoking 50k people without
being able to explain why (well, other than the key was compromised).

-Original Message-
From: dev-security-policy
<dev-security-policy-bounces+jeremy.rowley=digicert@lists.mozilla.org><mailto:dev-security-policy-bounces+jeremy.rowley=digicert@lists.mozilla.org>
On Behalf Of urijah--- via dev-security-policy
Sent: Wednesday, February 28, 2018 2:24 PM
To: 
mozilla-dev-security-pol...@lists.mozilla.org<mailto:mozilla-dev-security-pol...@lists.mozilla.org>
Subject: Re: How do you handle mass revocation requests?

Is Trustico's storage of private keys related to this security report from a
few months back (which did not appear to ever have been fully
investigated...)?

https://clicktime.symantec.com/a/1/gUJtIwxyRazHoxd9FIcS40cx3EykeGjrwv4urOpHj
u0=?d=-1kl6pTbi8aU0HfRh9ZNP86TCSEsjm5TvumKZsLlfO6tgKV5N4_MitDAK64FGwutmiXu3D
eTelMEfv3ep7X8kj6GIwMY8tUVM_WKGkqW_uhm0fHWgY9ZwlUytCHVYb_1WWceLMVi9w9Ec7h3sH
G1wIJHWNk2L8RcVycGdrIGRFvV76QXqOck9szIT1MqOPD6gSaSZRnxp-ItDtBU5xSit2RwQjv5sB
Ln2qFtnD9T_wuM6-KAeTUDm1O51uIY3NpT-q5hyIKG5YG-Ds6a9ceM0JgMrLhM3evdd-BmO3e6eP
1fXZXAQgJCZ_A7SnTbSagjzHzUVC65PN3AnrdJ6ANhZAwjjCdHdbjOeIqwInT-oFJ8lufSzaBeui
OECQuf-IqhCnfsam6KnTGT5k-aQlG_BjOW8fny0TS21Upa4k4aX2Xm7es0oRfOUS-OxAQLVKCGg-
cER_Pj3TmyKGPFFw_SzUky=https%3A%2F%2Fgroups.google.com<http://2Fgroups.google.com>%2Fd%2Fmsg%2Fmozilla
.dev.security.policy%2FCEww8w9q2zE%2FF_bzX1guCQAJ

Does Digicert have (or will it have) some sort of process in place to
prevent resellers from storing private keys so casually?

On Wednesday, February 28, 2018 at 12:38:16 PM UTC-5, Jeremy Rowley wrote:


Hi everyone,



I wanted to share an incident report regarding the revocation of
certain certificates ordered through a reseller.



On February 2nd, 2018, we received a request from Trustico to mass
revoke all certificates that had been ordered by end users through


Trustico.


Unfortunately, the email was not sent to the appropriate certificate
problem reporting channels and did not surface immediately so we're
delayed in sharing the concerns and information. Once we were alerted,
the team kicked off a debate that I wanted to bring to the CAB Forum.
Basically, our position is that resellers do not constitute
subscribers under the Baseline Requirement's definitions (Section
1.6.1). As such, we needed to confirm that either the key was
compromised or that they revocation was authorized by the domain
holder (the subscriber) prior to revoking the certificate. The


certificates were not alleged as compromised at that time.




Later, the company shared with us that they held the private keys and
the certificates were compromised, trying to trigger the BR's 24-hour
revocation requirement.  However, we insisted that the subscriber must
confirm the revocation request or there must be evidence of the private


key compromise.




Normally, we permit partners to revoke and manage the certificates
freely on behalf of their customer, with DigiCert providing all
validation and issuance services. However, the sheer number of
revocation requests (50k) and allegation of compromise triggered
concerns around the

Re: How do you handle mass revocation requests?

[Ryan Duff via dev-security-policy]

>From what I've read, it appears the situation here is that Trustico wanted to 
>revoke all their customer certs from Digicert so they could do a mass 
>migration to another CA (which is not a proper reason to revoke). When asked 
>for proof by Digicert that the certificates were compromised and needed to be 
>revoked, Trustico sent Digicert 23,000(!) private keys that *they had stored* 
>due to the fact that they were generated by their web-based system in order to 
>effectively *make them* compromised.

Am I missing anything?
___
dev-security-policy mailing list
dev-security-policy@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-security-policy

Re: How do you handle mass revocation requests?

[jomo via dev-security-policy]

> Basically, we're revoking 50k people without
> being able to explain why (well, other than the key was compromised). 

Unless I misunderstood, you originally said you received 23k compromised
keys and are revoking those.

> Currently, we are only revoking the certificates if we received the private
> keys. There are additional certificates the reseller requested to have
> revoked, but DigiCert has decided to disregard that request until we receive
> proof of compromise or more information about the cause of this incident.
Has DigiCert received proof of compromise of all 50k in the meantime?


On 28.2.18 22:42, Jeremy Rowley via dev-security-policy wrote:
> We don't have a process to prevent third parties from storing private keys.
> I'm not sure how that would even work considering the approved third-party
> use cases vs. non-approved use cases.  In fact, I'd postulate there's
> nothing wrong with Trustico holding the private keys if they were hosting
> the site or providing CDN services for all of these sites. The issue is
> Trustico alleged compromise of the certificates and sent us the private keys
> believed compromised in support. There were a lot of them. 
>
> This is a mass revocation without any explanation of what went wrong or why.
> The reseller mentioned and proved compromise, but there's no way to see if
> what happened, whether the issue was mitigated, or how it's going to be
> prevented from happening again. Basically, we're revoking 50k people without
> being able to explain why (well, other than the key was compromised). 
>
> -Original Message-
> From: dev-security-policy
> <dev-security-policy-bounces+jeremy.rowley=digicert@lists.mozilla.org>
> On Behalf Of urijah--- via dev-security-policy
> Sent: Wednesday, February 28, 2018 2:24 PM
> To: mozilla-dev-security-pol...@lists.mozilla.org
> Subject: Re: How do you handle mass revocation requests?
>
> Is Trustico's storage of private keys related to this security report from a
> few months back (which did not appear to ever have been fully
> investigated...)?
>
> https://clicktime.symantec.com/a/1/gUJtIwxyRazHoxd9FIcS40cx3EykeGjrwv4urOpHj
> u0=?d=-1kl6pTbi8aU0HfRh9ZNP86TCSEsjm5TvumKZsLlfO6tgKV5N4_MitDAK64FGwutmiXu3D
> eTelMEfv3ep7X8kj6GIwMY8tUVM_WKGkqW_uhm0fHWgY9ZwlUytCHVYb_1WWceLMVi9w9Ec7h3sH
> G1wIJHWNk2L8RcVycGdrIGRFvV76QXqOck9szIT1MqOPD6gSaSZRnxp-ItDtBU5xSit2RwQjv5sB
> Ln2qFtnD9T_wuM6-KAeTUDm1O51uIY3NpT-q5hyIKG5YG-Ds6a9ceM0JgMrLhM3evdd-BmO3e6eP
> 1fXZXAQgJCZ_A7SnTbSagjzHzUVC65PN3AnrdJ6ANhZAwjjCdHdbjOeIqwInT-oFJ8lufSzaBeui
> OECQuf-IqhCnfsam6KnTGT5k-aQlG_BjOW8fny0TS21Upa4k4aX2Xm7es0oRfOUS-OxAQLVKCGg-
> cER_Pj3TmyKGPFFw_SzUky=https%3A%2F%2Fgroups.google.com%2Fd%2Fmsg%2Fmozilla
> .dev.security.policy%2FCEww8w9q2zE%2FF_bzX1guCQAJ
>
> Does Digicert have (or will it have) some sort of process in place to
> prevent resellers from storing private keys so casually?
>
> On Wednesday, February 28, 2018 at 12:38:16 PM UTC-5, Jeremy Rowley wrote:
>> Hi everyone,
>>
>>  
>>
>> I wanted to share an incident report regarding the revocation of 
>> certain certificates ordered through a reseller.
>>
>>  
>>
>> On February 2nd, 2018, we received a request from Trustico to mass 
>> revoke all certificates that had been ordered by end users through
> Trustico.
>> Unfortunately, the email was not sent to the appropriate certificate 
>> problem reporting channels and did not surface immediately so we're 
>> delayed in sharing the concerns and information. Once we were alerted, 
>> the team kicked off a debate that I wanted to bring to the CAB Forum. 
>> Basically, our position is that resellers do not constitute 
>> subscribers under the Baseline Requirement's definitions (Section 
>> 1.6.1). As such, we needed to confirm that either the key was 
>> compromised or that they revocation was authorized by the domain 
>> holder (the subscriber) prior to revoking the certificate. The
> certificates were not alleged as compromised at that time.
>>  
>>
>> Later, the company shared with us that they held the private keys and 
>> the certificates were compromised, trying to trigger the BR's 24-hour 
>> revocation requirement.  However, we insisted that the subscriber must 
>> confirm the revocation request or there must be evidence of the private
> key compromise.
>>  
>>
>> Normally, we permit partners to revoke and manage the certificates 
>> freely on behalf of their customer, with DigiCert providing all 
>> validation and issuance services. However, the sheer number of 
>> revocation requests (50k) and allegation of compromise triggered 
>> concerns around the impact to the web and browser users.

RE: How do you handle mass revocation requests?

[Jeremy Rowley via dev-security-policy]

Yep - that was you. Thanks a ton. We posted 10 CSRs so far. Is this what you
were thinking? 

-Original Message-
From: Nick Lamb <n...@tlrmx.org> 
Sent: Wednesday, February 28, 2018 2:37 PM
To: dev-security-policy@lists.mozilla.org
Cc: Jeremy Rowley <jeremy.row...@digicert.com>
Subject: Re: How do you handle mass revocation requests?

On Wed, 28 Feb 2018 20:03:51 +
Jeremy Rowley via dev-security-policy
<dev-security-policy@lists.mozilla.org> wrote:

> The keys were emailed to me. I'm trying to get a project together 
> where we self-sign a cert with each of the keys and publish them.
> That way there's evidence to the community of the compromise without 
> simply listing 23k private keys. Someone on Reddit suggested that, 
> which I really appreciated.

That's probably me (tialaramex).

Anyway, if it is me you're referring to, I suggested using the private keys
to issue a bogus CSR. CSRs are signed, proving that whoever made them had
the corresponding private key but they avoid the confusion that comes from
DigiCert (or its employees) issuing bogus certs.
Everybody reading m.d.s.policy can still see that a self-signed cert is
harmless and not an attack, but it may be harder to explain in a soundbite.
Maybe more technically able contributors disagree ?


smime.p7s
Description: S/MIME cryptographic signature
___
dev-security-policy mailing list
dev-security-policy@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-security-policy

RE: How do you handle mass revocation requests?

[Jeremy Rowley via dev-security-policy]

We don't have a process to prevent third parties from storing private keys.
I'm not sure how that would even work considering the approved third-party
use cases vs. non-approved use cases.  In fact, I'd postulate there's
nothing wrong with Trustico holding the private keys if they were hosting
the site or providing CDN services for all of these sites. The issue is
Trustico alleged compromise of the certificates and sent us the private keys
believed compromised in support. There were a lot of them. 

This is a mass revocation without any explanation of what went wrong or why.
The reseller mentioned and proved compromise, but there's no way to see if
what happened, whether the issue was mitigated, or how it's going to be
prevented from happening again. Basically, we're revoking 50k people without
being able to explain why (well, other than the key was compromised). 

-Original Message-
From: dev-security-policy
<dev-security-policy-bounces+jeremy.rowley=digicert@lists.mozilla.org>
On Behalf Of urijah--- via dev-security-policy
Sent: Wednesday, February 28, 2018 2:24 PM
To: mozilla-dev-security-pol...@lists.mozilla.org
Subject: Re: How do you handle mass revocation requests?

Is Trustico's storage of private keys related to this security report from a
few months back (which did not appear to ever have been fully
investigated...)?

https://clicktime.symantec.com/a/1/gUJtIwxyRazHoxd9FIcS40cx3EykeGjrwv4urOpHj
u0=?d=-1kl6pTbi8aU0HfRh9ZNP86TCSEsjm5TvumKZsLlfO6tgKV5N4_MitDAK64FGwutmiXu3D
eTelMEfv3ep7X8kj6GIwMY8tUVM_WKGkqW_uhm0fHWgY9ZwlUytCHVYb_1WWceLMVi9w9Ec7h3sH
G1wIJHWNk2L8RcVycGdrIGRFvV76QXqOck9szIT1MqOPD6gSaSZRnxp-ItDtBU5xSit2RwQjv5sB
Ln2qFtnD9T_wuM6-KAeTUDm1O51uIY3NpT-q5hyIKG5YG-Ds6a9ceM0JgMrLhM3evdd-BmO3e6eP
1fXZXAQgJCZ_A7SnTbSagjzHzUVC65PN3AnrdJ6ANhZAwjjCdHdbjOeIqwInT-oFJ8lufSzaBeui
OECQuf-IqhCnfsam6KnTGT5k-aQlG_BjOW8fny0TS21Upa4k4aX2Xm7es0oRfOUS-OxAQLVKCGg-
cER_Pj3TmyKGPFFw_SzUky=https%3A%2F%2Fgroups.google.com%2Fd%2Fmsg%2Fmozilla
.dev.security.policy%2FCEww8w9q2zE%2FF_bzX1guCQAJ

Does Digicert have (or will it have) some sort of process in place to
prevent resellers from storing private keys so casually?

On Wednesday, February 28, 2018 at 12:38:16 PM UTC-5, Jeremy Rowley wrote:
> Hi everyone,
> 
>  
> 
> I wanted to share an incident report regarding the revocation of 
> certain certificates ordered through a reseller.
> 
>  
> 
> On February 2nd, 2018, we received a request from Trustico to mass 
> revoke all certificates that had been ordered by end users through
Trustico.
> Unfortunately, the email was not sent to the appropriate certificate 
> problem reporting channels and did not surface immediately so we're 
> delayed in sharing the concerns and information. Once we were alerted, 
> the team kicked off a debate that I wanted to bring to the CAB Forum. 
> Basically, our position is that resellers do not constitute 
> subscribers under the Baseline Requirement's definitions (Section 
> 1.6.1). As such, we needed to confirm that either the key was 
> compromised or that they revocation was authorized by the domain 
> holder (the subscriber) prior to revoking the certificate. The
certificates were not alleged as compromised at that time.
> 
>  
> 
> Later, the company shared with us that they held the private keys and 
> the certificates were compromised, trying to trigger the BR's 24-hour 
> revocation requirement.  However, we insisted that the subscriber must 
> confirm the revocation request or there must be evidence of the private
key compromise.
> 
>  
> 
> Normally, we permit partners to revoke and manage the certificates 
> freely on behalf of their customer, with DigiCert providing all 
> validation and issuance services. However, the sheer number of 
> revocation requests (50k) and allegation of compromise triggered 
> concerns around the impact to the web and browser users. Therefore, 
> this was categorized as high risk, especially considering the relationship
between Trustico and DigiCert is terminating.
> 
>  
> 
> On 2/27/2018, at my request for proof of compromise, we received a 
> file with 23k private keys matched to specific Trustico customers. 
> This definitely triggered our 24-hour revocation processing requirement
under 4.9.1.1.3.
> Once we received the keys, we confirmed that these were indeed the 
> matching private keys for the reported certificates. We will be 
> revoking these certificates today (February 28th, 2018).
> 
>  
> 
> At this time, Trustico has not provided any information about how 
> these certificates were compromised or how they acquired the private 
> keys. As is standard practice for a Certificate Authority, DigiCert 
> never had possession of these private keys.
> 
>  
> 
> Currently, we are only revoking the certificates if we received the 
> private keys. There are 

Re: How do you handle mass revocation requests?

[Ryan Sleevi via dev-security-policy]

On Wed, Feb 28, 2018 at 4:23 PM, urijah--- via dev-security-policy <
dev-security-policy@lists.mozilla.org> wrote:

> Is Trustico's storage of private keys related to this security report from
> a few months back (which did not appear to ever have been fully
> investigated...)?
>

It was fully investigated. There's no evidence to suggest relation, other
than there's no way to effectively make rules on that (other than the
threat of revocation for breach of contract, which you can only demonstrate
when you can prove it, which you'd revoke anyways)


>
> https://groups.google.com/d/msg/mozilla.dev.security.policy/CEww8w9q2zE/F_
> bzX1guCQAJ
>
> Does Digicert have (or will it have) some sort of process in place to
> prevent resellers from storing private keys so casually?


While well-intentioned, I don't think this is really a reasonable thing to
ask/expect. If they had contractual requirements on resellers, it just
means you'd have new entities spring up that are equivalent to resellers
without the requirements.

Would we similarly require that hosting providers don't store the private
keys? It's the same conceptual issue.
___
dev-security-policy mailing list
dev-security-policy@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-security-policy

Re: How do you handle mass revocation requests?

[Nick Lamb via dev-security-policy]

On Wed, 28 Feb 2018 20:03:51 +
Jeremy Rowley via dev-security-policy
 wrote:

> The keys were emailed to me. I'm trying to get a project together
> where we self-sign a cert with each of the keys and publish them.
> That way there's evidence to the community of the compromise without
> simply listing 23k private keys. Someone on Reddit suggested that,
> which I really appreciated.

That's probably me (tialaramex).

Anyway, if it is me you're referring to, I suggested using the private
keys to issue a bogus CSR. CSRs are signed, proving that whoever made
them had the corresponding private key but they avoid the confusion
that comes from DigiCert (or its employees) issuing bogus certs.
Everybody reading m.d.s.policy can still see that a self-signed cert is
harmless and not an attack, but it may be harder to explain in a
soundbite. Maybe more technically able contributors disagree ?
___
dev-security-policy mailing list
dev-security-policy@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-security-policy

Re: How do you handle mass revocation requests?

[urijah--- via dev-security-policy]

Is Trustico's storage of private keys related to this security report from a 
few months back (which did not appear to ever have been fully investigated...)?

https://groups.google.com/d/msg/mozilla.dev.security.policy/CEww8w9q2zE/F_bzX1guCQAJ

Does Digicert have (or will it have) some sort of process in place to prevent 
resellers from storing private keys so casually?

On Wednesday, February 28, 2018 at 12:38:16 PM UTC-5, Jeremy Rowley wrote:
> Hi everyone,
> 
>  
> 
> I wanted to share an incident report regarding the revocation of certain
> certificates ordered through a reseller.
> 
>  
> 
> On February 2nd, 2018, we received a request from Trustico to mass revoke
> all certificates that had been ordered by end users through Trustico.
> Unfortunately, the email was not sent to the appropriate certificate problem
> reporting channels and did not surface immediately so we're delayed in
> sharing the concerns and information. Once we were alerted, the team kicked
> off a debate that I wanted to bring to the CAB Forum. Basically, our
> position is that resellers do not constitute subscribers under the Baseline
> Requirement's definitions (Section 1.6.1). As such, we needed to confirm
> that either the key was compromised or that they revocation was authorized
> by the domain holder (the subscriber) prior to revoking the certificate. The
> certificates were not alleged as compromised at that time.  
> 
>  
> 
> Later, the company shared with us that they held the private keys and the
> certificates were compromised, trying to trigger the BR's 24-hour revocation
> requirement.  However, we insisted that the subscriber must confirm the
> revocation request or there must be evidence of the private key compromise. 
> 
>  
> 
> Normally, we permit partners to revoke and manage the certificates freely on
> behalf of their customer, with DigiCert providing all validation and
> issuance services. However, the sheer number of revocation requests (50k)
> and allegation of compromise triggered concerns around the impact to the web
> and browser users. Therefore, this was categorized as high risk, especially
> considering the relationship between Trustico and DigiCert is terminating.
> 
>  
> 
> On 2/27/2018, at my request for proof of compromise, we received a file with
> 23k private keys matched to specific Trustico customers. This definitely
> triggered our 24-hour revocation processing requirement under 4.9.1.1.3.
> Once we received the keys, we confirmed that these were indeed the matching
> private keys for the reported certificates. We will be revoking these
> certificates today (February 28th, 2018).
> 
>  
> 
> At this time, Trustico has not provided any information about how these
> certificates were compromised or how they acquired the private keys. As is
> standard practice for a Certificate Authority, DigiCert never had possession
> of these private keys.
> 
>  
> 
> Currently, we are only revoking the certificates if we received the private
> keys. There are additional certificates the reseller requested to have
> revoked, but DigiCert has decided to disregard that request until we receive
> proof of compromise or more information about the cause of this incident.
> 
>  
> 
> DigiCert sent out emails to the affected customers in order to notify them
> that their certificate(s) are being revoked. This revocation only affects
> those customers and there is no additional exposure that we are aware of at
> this time, nor any reason to believe there is.  
> 
>  
> 
> This raises a question about the MDSP policy and CAB Forum requirements. Who
> is the subscriber in the reseller relation?  We believe this to be the key
> holder. However, the language is unclear. I think we followed the letter and
> spirit of the BRs here, but I'd like feedback, perhaps leading to a ballot
> that clarifies the subscriber in a reseller relationship.
> 
>  
> 
> This also brings up a ballot about the level of due diligence required for
> cert revocation. We generally believe that the private key or demonstration
> of domain control is sufficient to request revocation. Others are at the CAs
> discretion. Should we clarify what the due diligence looks like? Are there
> other things we should have done or been doing? 
> 
>  
> 
> What kind of transparency would the Mozilla community like around this
> issue? There aren't many more facts than I shared above, but there is a lot
> of speculation. Let me know what I can share to help alleviate confusion and
> answer questions.
> 
>  
> 
> Jeremy

___
dev-security-policy mailing list
dev-security-policy@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-security-policy

RE: How do you handle mass revocation requests?

[kevin.beaumont--- via dev-security-policy]

It’s absolutely incredible that Trustico has 23k private keys, and just 
attached them to an email. This suggests serious flaws in the CA/reseller 
relationship.
___
dev-security-policy mailing list
dev-security-policy@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-security-policy

Re: How do you handle mass revocation requests?

[Matthew Hardeman via dev-security-policy]

Did this whole thing start because someone at Trustico wanted to accelerate the 
process of getting their resold Symantec certificates reissued under a DigiCert 
trust path?

And somehow some misinformed soul imagined creating a revocation crisis would 
somehow help achieve that  goal without significant consequences?

If so, that surfaces a significant potential problem with the reseller 
ecosystem, which as I understand it has largely gone unaudited as yet 
(providing the CA actually provides the validation functions).  It shows what 
havoc a reseller can cause from poor judgement.

Then there's the issue of a reseller holding onto the private keys 
indefinitely...
___
dev-security-policy mailing list
dev-security-policy@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-security-policy

Re: How do you handle mass revocation requests?

[Matthew Hardeman via dev-security-policy]

I would echo Mr. Gaynor's point.

While it's perhaps a pedantic distinction, the private keys are definitely
compromised now and were the moment that Trustico provided the keys to
Digicert, even if Trustico is defined to be the original authorized
recipient.

The CA is explicitly not to be in possession of these private keys and if
Digicert's assertions are correct, Digicert now holds the private keys.

It would seem that revocation is unquestionably the appropriate remedy at
this time, despite whether or not Trustico understands what they requested.

On Wed, Feb 28, 2018 at 1:32 PM, Alex Gaynor via dev-security-policy <
dev-security-policy@lists.mozilla.org> wrote:

> I would say that at the point that Trustico emailed them to DigiCert they
> necessarily became compromised -- while Trustico may (or may not) have been
> authorized to escrowing the keys by the subscriber, the subscriber did not
> authorize them to be emailed around, presumably.
>
> Alex
>
> On Wed, Feb 28, 2018 at 2:29 PM, Wayne Thayer via dev-security-policy <
> dev-security-policy@lists.mozilla.org> wrote:
>
> > On Wed, Feb 28, 2018 at 12:13 PM, timx84039--- via dev-security-policy <
> > dev-security-policy@lists.mozilla.org> wrote:
> >
> > >
> > > Regarding to our investigation they were only able to send the private
> > > keys for those certificates where the CSR / private key pair were
> > generated
> > > within their online private key generating tool. This has to be the 23k
> > > amount of keys which Jeremy received.
> > >
> > > I am not aware of guidelines of the CA/B forum but keeping 23.000 (!)
> > > private keys at your online platform seems more than alarming and is
> > > careless and the public should be made aware of this fact.
> > >
> > > I agree with this sentiment, but I also think it creates another policy
> > question with respect to DigiCert's decision to revoke due to key
> > compromise: were these 23,000 keys really compromised? The BR definition
> of
> > Key Compromise is:
> >
> > A Private Key is said to be compromised if its value has been disclosed
> to
> > an unauthorized person, an unauthorized person has had access to it, or
> > there exists a practical technique by which an unauthorized person may
> > discover its value. A Private Key is also considered compromised if
> methods
> > have been developed that can easily calculate it based on the Public Key
> > (such as a Debian weak key, see http://wiki.debian.org/SSLkeys) or if
> > there
> > is clear evidence that the specific method used to generate the Private
> Key
> > was flawed.
> >
> > In this case it might be reasonable to argue that Trustico was
> unauthorized
> > (unless their customers agreed to key escrow when using the online key
> > generation tool). However, in the case of a hosting provider reselling
> > certificates for use on their platform, it's required that they hold the
> > private key and we don't consider that a Key Compromise.
> >
> > - Wayne
> > ___
> > dev-security-policy mailing list
> > dev-security-policy@lists.mozilla.org
> > https://lists.mozilla.org/listinfo/dev-security-policy
> >
> ___
> dev-security-policy mailing list
> dev-security-policy@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-security-policy
>
___
dev-security-policy mailing list
dev-security-policy@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-security-policy

Re: How do you handle mass revocation requests?

[Ryan Hurst via dev-security-policy]

On Wednesday, February 28, 2018 at 11:56:04 AM UTC-8, Ryan Sleevi wrote:
> Assuming Trustico sent the keys to DigiCert, it definitely sounds like even
> if Trustico was authorized to hold the keys (which is a troubling argument,
> given all things), they themselves compromised the keys of their customers,
> and revocation is both correct and necessary. That is, whether or not
> Trustico believed they were compromised before, they compromised their
> customers keys by sending them, and it's both correct and accurate to
> notify the Subscribers that their keys have been compromised by their
> Reseller.

That seems to be the case to me as well.

It also seems that this situation should result in the UAs and/or CABFORUM 
re0visit section 6.1.2 
(https://github.com/cabforum/documents/blob/master/docs/BR.md) in the BRs.

Specifically, this section states:

```
Parties other than the Subscriber SHALL NOT archive the Subscriber Private Key 
without authorization by the Subscriber.

If the CA or any of its designated RAs generated the Private Key on behalf of 
the Subscriber, then the CA SHALL encrypt the Private Key for transport to the 
Subscriber.
```

In this case, TrustIco is not the subscriber, and there is no indication in 
their terms and conditions 
(https://www.trustico.com/terms/terms-and-conditions.php) that they are 
authorized to archive the private key. Yet clearly if they were able to provide 
20k+ private keys to DigiCert they are archiving them. This text seems to cover 
this case clearly but as worded I do not see how audits would catch this 
behavior. I think it may make sense for the CAs to be responsible for 
demonstrating how they and other non-subscribers in the lifecycle flow handle 
this case.

Additionally, it seems if the private keys were provided to DigiCert in a way 
they were verifiable by them they may have been stored in a non-encrypted 
fashion, at a minimum they were likley not generated and protected on an HSM. 
The BRs should probably be revised to specify some minimum level of security to 
be provided in these cases of for these cases to be simply disallowed 
altogether.

Finally, the associated text speaks to RAs but not to the non-subscriber 
(reseller) case, this gap should be addressed minimally.
___
dev-security-policy mailing list
dev-security-policy@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-security-policy

RE: How do you handle mass revocation requests?

[Jeremy Rowley via dev-security-policy]

The keys were emailed to me. I'm trying to get a project together where we
self-sign a cert with each of the keys and publish them. That way there's
evidence to the community of the compromise without simply listing 23k
private keys. Someone on Reddit suggested that, which I really appreciated.
I think the main hold up in this plan is, which websites do we want to call
out? 

-Original Message-
From: dev-security-policy
<dev-security-policy-bounces+jeremy.rowley=digicert@lists.mozilla.org>
On Behalf Of Peter Bowen via dev-security-policy
Sent: Wednesday, February 28, 2018 12:38 PM
To: Wayne Thayer <wtha...@mozilla.com>
Cc: timx84...@gmail.com; mozilla-dev-security-policy
<mozilla-dev-security-pol...@lists.mozilla.org>
Subject: Re: How do you handle mass revocation requests?

On Wed, Feb 28, 2018 at 11:29 AM, Wayne Thayer via dev-security-policy
<dev-security-policy@lists.mozilla.org> wrote:
> On Wed, Feb 28, 2018 at 12:13 PM, timx84039--- via dev-security-policy
<dev-security-policy@lists.mozilla.org> wrote:
>
>>
>> Regarding to our investigation they were only able to send the 
>> private keys for those certificates where the CSR / private key pair 
>> were generated within their online private key generating tool. This 
>> has to be the 23k amount of keys which Jeremy received.
>>
>> I am not aware of guidelines of the CA/B forum but keeping 23.000 (!) 
>> private keys at your online platform seems more than alarming and is 
>> careless and the public should be made aware of this fact.
>>
> I agree with this sentiment, but I also think it creates another 
> policy question with respect to DigiCert's decision to revoke due to 
> key
> compromise: were these 23,000 keys really compromised? The BR 
> definition of Key Compromise is:
>
> A Private Key is said to be compromised if its value has been 
> disclosed to an unauthorized person, an unauthorized person has had 
> access to it, or there exists a practical technique by which an 
> unauthorized person may discover its value. A Private Key is also 
> considered compromised if methods have been developed that can easily 
> calculate it based on the Public Key (such as a Debian weak key, see 
> http://wiki.debian.org/SSLkeys) or if there is clear evidence that the 
> specific method used to generate the Private Key was flawed.
>
> In this case it might be reasonable to argue that Trustico was 
> unauthorized (unless their customers agreed to key escrow when using 
> the online key generation tool). However, in the case of a hosting 
> provider reselling certificates for use on their platform, it's 
> required that they hold the private key and we don't consider that a Key
Compromise.

Jeremy's email suggests that the keys were emailed to him.  If this is
accurate, then it is reasonable that they have been "disclosed to an
unauthorized person".  The only other alternative, again assuming Jeremy did
receive the keys, is to determine that he was authorized by the subscriber
to access the keys.

Thanks,
Peter
___
dev-security-policy mailing list
dev-security-policy@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-security-policy


smime.p7s
Description: S/MIME cryptographic signature
___
dev-security-policy mailing list
dev-security-policy@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-security-policy

Re: How do you handle mass revocation requests?

[Ryan Sleevi via dev-security-policy]

On Wed, Feb 28, 2018 at 2:40 PM, Jeremy Rowley via dev-security-policy <
dev-security-policy@lists.mozilla.org> wrote:

> The end user agreed to the subscriber agreement, not Trustico. Our
> analysis follows what Peter B. posted – the subscriber is the “natural
> person or Legal Entity to whom a Certificate is issued and who is legally
> bound by a Subscriber Agreement or Terms of Use"—which in this case was
> Trustico’s customers.  In addition, we felt that given (1) the number of
> certificates Trustico was asking us to mass-revoke and (2) the lack of any
> substantiation of why Trustico thought the certs were “compromised,” we
> needed more information before revoking. At the minimum, it warranted
> alerting the contact for each certificate that revocation was imminent.
>

Assuming Trustico sent the keys to DigiCert, it definitely sounds like even
if Trustico was authorized to hold the keys (which is a troubling argument,
given all things), they themselves compromised the keys of their customers,
and revocation is both correct and necessary. That is, whether or not
Trustico believed they were compromised before, they compromised their
customers keys by sending them, and it's both correct and accurate to
notify the Subscribers that their keys have been compromised by their
Reseller.

If your Reseller compromises your keys capriciously,  it sounds like it
might be time to find a new Reseller.

I also agree that there’s no problem with the way or that the keys were
> provided to DigiCert for cert revocation. I certainly don’t want to
> discourage revocation of compromised certs!  My main question is how do you
> handle these things? The process at scale should not be different if a
> reseller requests revocation compared to any other security researcher. The
> question is how you handle the this volume of revocations when its happen?
> I’ve never received a revocation request of this magnitude before so I’m
> seeking the wisdom of the community in what we do.
>

For the remaining certificates, it sounds like, based on the evidence, that
Trustico has no authority to request revocation for those remaining, and it
would be monumentally hostile/unwise of them to subject that their
customers to do that.

To the more general case - how to handle mass revocations - this seems
similar to when Heartbleed hit, and the lessons from then apply now:

- CAs should design their systems and be prepared for mass revocations on
sufficient demonstration
  - Sharding out CRLs, for example, to ensure no CRL becomes unreasonably
large (say, >64KB, which is many clients' limits), using critical
issuingDistributionPoint extensions on the CRLs and hosting at unique URLs
  - Designing the OCSP serving infrastructure to be able to handle a large
influx of revocations
  - Reducing the overall lifetime of certificates to minimize the total
number of ongoing (OCSP, CRL) signatures that would need to be produced

- Site Operators should design their systems to be prepared for sudden
revocation
  - Don't introduce unnecessary third parties (such as Resellers), when
possible
  - Invest in automation such that certificate replacement is
automated/automatic
  - Corollary: Have multiple CAs support the automation mechanisms used
(such as standard automation methods) in the event it is the CA compromised
  - Corollary: Support multiple automation validation methods (such as DNS
+ HTTP + TLS) in the event it is the automation method that is compromised
___
dev-security-policy mailing list
dev-security-policy@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-security-policy

RE: How do you handle mass revocation requests?

[Jeremy Rowley via dev-security-policy]

The end user agreed to the subscriber agreement, not Trustico. Our analysis 
follows what Peter B. posted – the subscriber is the “natural person or Legal 
Entity to whom a Certificate is issued and who is legally bound by a Subscriber 
Agreement or Terms of Use"—which in this case was Trustico’s customers.  In 
addition, we felt that given (1) the number of certificates Trustico was asking 
us to mass-revoke and (2) the lack of any substantiation of why Trustico 
thought the certs were “compromised,” we needed more information before 
revoking. At the minimum, it warranted alerting the contact for each 
certificate that revocation was imminent.

 

I also agree that there’s no problem with the way or that the keys were 
provided to DigiCert for cert revocation. I certainly don’t want to discourage 
revocation of compromised certs!  My main question is how do you handle these 
things? The process at scale should not be different if a reseller requests 
revocation compared to any other security researcher. The question is how you 
handle the this volume of revocations when its happen? I’ve never received a 
revocation request of this magnitude before so I’m seeking the wisdom of the 
community in what we do.

 

I’m happy to share any of the details I can. 

 

Jeremy

 

 

From: Ryan Sleevi <r...@sleevi.com> 
Sent: Wednesday, February 28, 2018 11:58 AM
To: Jeremy Rowley <jeremy.row...@digicert.com>
Cc: mozilla-dev-security-pol...@lists.mozilla.org
Subject: Re: How do you handle mass revocation requests?

 

 

 

On Wed, Feb 28, 2018 at 12:37 PM, Jeremy Rowley via dev-security-policy 
<dev-security-policy@lists.mozilla.org 
<mailto:dev-security-policy@lists.mozilla.org> > wrote:

On February 2nd, 2018, we received a request from Trustico to mass revoke
all certificates that had been ordered by end users through Trustico.
Unfortunately, the email was not sent to the appropriate certificate problem
reporting channels and did not surface immediately so we're delayed in
sharing the concerns and information. Once we were alerted, the team kicked
off a debate that I wanted to bring to the CAB Forum. Basically, our
position is that resellers do not constitute subscribers under the Baseline
Requirement's definitions (Section 1.6.1). As such, we needed to confirm
that either the key was compromised or that they revocation was authorized
by the domain holder (the subscriber) prior to revoking the certificate. The
certificates were not alleged as compromised at that time.

 

I think there's a little nuance to this in the general case (e.g. consider 
"Resllers" who are also the hosting provider, and thus constitute the 
Applicant/Subscriber even when they're not the domain holder, but authorized by 
them), but based on this specific case, I think this sounds like a correct 
determination.

 

I think the biggest question is who agreed to the terms - Trustico or the 
end-user - and that mostly impacts the rest of the decision here. Further, did 
Trustico warrant on behalf of the user that the user agreed to the terms (in 
which case, I would think it's a bit of a copout, and it's really Trustico 
agreeing, thus the Subscriber), or did DigiCert have direct communication with 
the user on the basis of Trustico's introduction (in which case, yeah, the 
Subscriber is the user)

 

Anyway, just highlighting it here as perhaps not a uniform consensus, but 
perhaps not as material as what follows.

 

On 2/27/2018, at my request for proof of compromise, we received a file with
23k private keys matched to specific Trustico customers. This definitely
triggered our 24-hour revocation processing requirement under 4.9.1.1.3.
Once we received the keys, we confirmed that these were indeed the matching
private keys for the reported certificates. We will be revoking these
certificates today (February 28th, 2018).

 

I think all of this sounds reasonable, no different than a security researcher 
(or random member of the public) who were to claim compromise with no evidence 
of that.

 

Currently, we are only revoking the certificates if we received the private
keys. There are additional certificates the reseller requested to have
revoked, but DigiCert has decided to disregard that request until we receive
proof of compromise or more information about the cause of this incident.

 

For the same reason that "Jane Random User" should not be able to cause 
revocation merely by assertion, I think that sounds reasonable.

 

This raises a question about the MDSP policy and CAB Forum requirements. Who
is the subscriber in the reseller relation?  We believe this to be the key
holder. However, the language is unclear. I think we followed the letter and
spirit of the BRs here, but I'd like feedback, perhaps leading to a ballot
that clarifies the subscriber in a reseller relationship.

 

I think the question here is less about who is the Applicant, but who is the 
Applicant Representative

Re: How do you handle mass revocation requests?

[Peter Bowen via dev-security-policy]

On Wed, Feb 28, 2018 at 11:29 AM, Wayne Thayer via dev-security-policy
 wrote:
> On Wed, Feb 28, 2018 at 12:13 PM, timx84039--- via dev-security-policy 
>  wrote:
>
>>
>> Regarding to our investigation they were only able to send the private
>> keys for those certificates where the CSR / private key pair were generated
>> within their online private key generating tool. This has to be the 23k
>> amount of keys which Jeremy received.
>>
>> I am not aware of guidelines of the CA/B forum but keeping 23.000 (!)
>> private keys at your online platform seems more than alarming and is
>> careless and the public should be made aware of this fact.
>>
> I agree with this sentiment, but I also think it creates another policy
> question with respect to DigiCert's decision to revoke due to key
> compromise: were these 23,000 keys really compromised? The BR definition of
> Key Compromise is:
>
> A Private Key is said to be compromised if its value has been disclosed to
> an unauthorized person, an unauthorized person has had access to it, or
> there exists a practical technique by which an unauthorized person may
> discover its value. A Private Key is also considered compromised if methods
> have been developed that can easily calculate it based on the Public Key
> (such as a Debian weak key, see http://wiki.debian.org/SSLkeys) or if there
> is clear evidence that the specific method used to generate the Private Key
> was flawed.
>
> In this case it might be reasonable to argue that Trustico was unauthorized
> (unless their customers agreed to key escrow when using the online key
> generation tool). However, in the case of a hosting provider reselling
> certificates for use on their platform, it's required that they hold the
> private key and we don't consider that a Key Compromise.

Jeremy's email suggests that the keys were emailed to him.  If this is
accurate, then it is reasonable that they have been "disclosed to an
unauthorized person".  The only other alternative, again assuming
Jeremy did receive the keys, is to determine that he was authorized by
the subscriber to access the keys.

Thanks,
Peter
___
dev-security-policy mailing list
dev-security-policy@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-security-policy

Re: How do you handle mass revocation requests?

[Alex Gaynor via dev-security-policy]

I would say that at the point that Trustico emailed them to DigiCert they
necessarily became compromised -- while Trustico may (or may not) have been
authorized to escrowing the keys by the subscriber, the subscriber did not
authorize them to be emailed around, presumably.

Alex

On Wed, Feb 28, 2018 at 2:29 PM, Wayne Thayer via dev-security-policy <
dev-security-policy@lists.mozilla.org> wrote:

> On Wed, Feb 28, 2018 at 12:13 PM, timx84039--- via dev-security-policy <
> dev-security-policy@lists.mozilla.org> wrote:
>
> >
> > Regarding to our investigation they were only able to send the private
> > keys for those certificates where the CSR / private key pair were
> generated
> > within their online private key generating tool. This has to be the 23k
> > amount of keys which Jeremy received.
> >
> > I am not aware of guidelines of the CA/B forum but keeping 23.000 (!)
> > private keys at your online platform seems more than alarming and is
> > careless and the public should be made aware of this fact.
> >
> > I agree with this sentiment, but I also think it creates another policy
> question with respect to DigiCert's decision to revoke due to key
> compromise: were these 23,000 keys really compromised? The BR definition of
> Key Compromise is:
>
> A Private Key is said to be compromised if its value has been disclosed to
> an unauthorized person, an unauthorized person has had access to it, or
> there exists a practical technique by which an unauthorized person may
> discover its value. A Private Key is also considered compromised if methods
> have been developed that can easily calculate it based on the Public Key
> (such as a Debian weak key, see http://wiki.debian.org/SSLkeys) or if
> there
> is clear evidence that the specific method used to generate the Private Key
> was flawed.
>
> In this case it might be reasonable to argue that Trustico was unauthorized
> (unless their customers agreed to key escrow when using the online key
> generation tool). However, in the case of a hosting provider reselling
> certificates for use on their platform, it's required that they hold the
> private key and we don't consider that a Key Compromise.
>
> - Wayne
> ___
> dev-security-policy mailing list
> dev-security-policy@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-security-policy
>
___
dev-security-policy mailing list
dev-security-policy@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-security-policy

Re: How do you handle mass revocation requests?

[Wayne Thayer via dev-security-policy]

On Wed, Feb 28, 2018 at 12:13 PM, timx84039--- via dev-security-policy <
dev-security-policy@lists.mozilla.org> wrote:

>
> Regarding to our investigation they were only able to send the private
> keys for those certificates where the CSR / private key pair were generated
> within their online private key generating tool. This has to be the 23k
> amount of keys which Jeremy received.
>
> I am not aware of guidelines of the CA/B forum but keeping 23.000 (!)
> private keys at your online platform seems more than alarming and is
> careless and the public should be made aware of this fact.
>
> I agree with this sentiment, but I also think it creates another policy
question with respect to DigiCert's decision to revoke due to key
compromise: were these 23,000 keys really compromised? The BR definition of
Key Compromise is:

A Private Key is said to be compromised if its value has been disclosed to
an unauthorized person, an unauthorized person has had access to it, or
there exists a practical technique by which an unauthorized person may
discover its value. A Private Key is also considered compromised if methods
have been developed that can easily calculate it based on the Public Key
(such as a Debian weak key, see http://wiki.debian.org/SSLkeys) or if there
is clear evidence that the specific method used to generate the Private Key
was flawed.

In this case it might be reasonable to argue that Trustico was unauthorized
(unless their customers agreed to key escrow when using the online key
generation tool). However, in the case of a hosting provider reselling
certificates for use on their platform, it's required that they hold the
private key and we don't consider that a Key Compromise.

- Wayne
___
dev-security-policy mailing list
dev-security-policy@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-security-policy

RE: How do you handle mass revocation requests?

[Jeremy Rowley via dev-security-policy]

I agree with the OV, EV, and IV. Admittedly, DV certs, which constitute almost 
all the certs, are relatively new to DigiCert so that's partly where the 
question arises. We identified it as the key holder or the domain holder. 
Hence, we'd revoke with confirmation of a domain validation. The reseller 
could be the subscriber, but I'm not sure how we tell with DV certs. This is 
especially with legacy Symantec customers where we are still trying to 
establish the personal relationship and understand their use cases, 
communication expectations, etc.

-Original Message-
From: Peter Bowen <pzbo...@gmail.com>
Sent: Wednesday, February 28, 2018 12:14 PM
To: Jeremy Rowley <jeremy.row...@digicert.com>
Cc: mozilla-dev-security-pol...@lists.mozilla.org
Subject: Re: How do you handle mass revocation requests?

On Wed, Feb 28, 2018 at 9:37 AM, Jeremy Rowley via dev-security-policy 
<dev-security-policy@lists.mozilla.org> wrote:
> Once we were alerted, the team kicked
> off a debate that I wanted to bring to the CAB Forum. Basically, our
> position is that resellers do not constitute subscribers under the
> Baseline Requirement's definitions (Section 1.6.1). As such, we needed
> to confirm that either the key was compromised or that they revocation
> was authorized by the domain holder (the subscriber) prior to revoking
> the certificate. The certificates were not alleged as compromised at that 
> time.

> This raises a question about the MDSP policy and CAB Forum
> requirements. Who is the subscriber in the reseller relation?  We
> believe this to be the key holder. However, the language is unclear. I
> think we followed the letter and spirit of the BRs here, but I'd like
> feedback, perhaps leading to a ballot that clarifies the subscriber in a 
> reseller relationship.

For certs with subject identity information (commonly called IV, OV, and EV 
certs), there is no question about the subscriber.  The Subscriber is the 
entity identified in the subject: "The Subject is either the Subscriber or a 
device under the control and operation of the Subscriber."

For certificates without subject identity information (DV certificates), the 
certificate does not list the subscriber.  However the CA clearly knows the 
subscriber, as the subscriber is the "natural person or Legal Entity to whom a 
Certificate is issued and who is legally bound by a Subscriber Agreement or 
Terms of Use"

In some cases the "reseller" might be the subscriber if the reseller is a 
hosting company and is the one that accepts the subscriber agreement but in 
the traditional reseller model their customer is the subscriber as the 
reseller's customer is the one accepting the subscriber agreement.

Given that DigiCert appears to have contact information for the Trustico 
customers, that suggests that the Trustico customer is likely the subscriber, 
but looking at IV/OV/EV certificates (if any) should tell for sure.

Thanks,
Peter


smime.p7s
Description: S/MIME cryptographic signature
___
dev-security-policy mailing list
dev-security-policy@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-security-policy

RE: How do you handle mass revocation requests?

[John Merrill via dev-security-policy]

Let's talk it through with Mike J as this will end up in court

-Original Message-
From: dev-security-policy 
[mailto:dev-security-policy-bounces+john.merrill=digicert@lists.mozilla.org]
 On Behalf Of Jeremy Rowley via dev-security-policy
Sent: Wednesday, February 28, 2018 12:20 PM
To: google.mana...@trustico.com; mozilla-dev-security-pol...@lists.mozilla.org
Subject: RE: How do you handle mass revocation requests?

I believe transparency is the best policy.  I think it'd be helpful to the 
community if we could post the email exchange about the revocation. We can 
redact the agreement termination portions if you'd like, but that'd give a lot 
more clarity around what's going on. Do I have your permission to post those 
emails?

-Original Message-
From: dev-security-policy 
<dev-security-policy-bounces+jeremy.rowley=digicert@lists.mozilla.org> On 
Behalf Of google.manager--- via dev-security-policy
Sent: Wednesday, February 28, 2018 11:54 AM
To: mozilla-dev-security-pol...@lists.mozilla.org
Subject: How do you handle mass revocation requests?

Jeremy,

Today many of our customers experienced lengthy delays when attempting to 
contact us via phone, e-mail and live chat. The reason for the delays were due 
to an unexpected e-mail that DigiCert sent to our customers containing some 
inaccurate information. We were not informed that the e-mail would be sent and 
were caught by surprise. We had disabled this e-mail within your control panel 
and opted to send our own notices, though you took it upon yourselves to send 
my personal details to 20,000 customers.

We didn't authorise DigiCert to contact our customers and we didn't approve the 
content of their e-mail. At no time had any private keys been compromised, nor 
had we ever informed to you that any private keys had been compromised. During 
our many discussions over the past week we put it to you that we believe 
Symantec to have operated our account in a manner whereby it had been 
compromised. Your usage of the word compromise has been twisted by you to your 
benefit and is absolutely defamatory.

We believe the orders placed via our Symantec account were at risk and were 
poorly managed. We have been questioning Symantec without response as to 
concerning items for about a year. Symantec simply ignored our concerns and 
appeared to bury them under the next issue that arose.

There are a range of issues that Trustico intends to investigate via legal 
means.

In good conscience we decided it wasn't ideal to have any active SSL 
Certificates on the Symantec systems, nor any that didn't meet our stringent 
security requirements. Our concerns also relate to the upcoming distrust of all 
Symantec SSL Certificate brands within Google Chrome and the reasoning behind 
it. The same management team responsible for that situation is duly employed at 
DigiCert and are fully managing our account, causing grave concern on our part 
as it appears to be business as usual with a new name. We were also a victim 
whereby Symantec mis-issued SSL Certificates owned by us, subsequently we were 
asked to keep the matter quiet, under a confidentially notice.

We had implemented a system to ensure that all customers would receive a 
replacement SSL Certificate, though today it had failed to perform this 
function.

In our view it is absolutely critical that an SSL Certificate performs its 
intended function. Symantec's issue with Google appeared to seal that deal, 
whereby they will all eventually fail due to distrust. In accordance with CAB 
Forum guidelines we acted to immediately revoke active SSL Certificates whereby 
trust was questionable.

Trustico absolutely distrusts the Symantec brand due to the issues that forced 
Symantec into having to hand over its entire authentication business to an 
alternate CA and a range of issues beforehand. Though, Symantec was ultimately 
acquired by DigiCert - though now DigiCert appears to be influenced by the 
Symantec management team - that to date still is managing our account. Trustico 
stopped offering the Symantec brands early February after a meeting with your 
Symantec management team, whereby they had disclosed to us that various 
reckless issues had occurred (recording available).

We realize that this mass revocation is bothersome and time consuming for all 
that have been affected. We're working to contact all customers to get orders 
replaced as priority and working through a backlog of enquiries. 

Unfortunately, things didn't go very well for us today and we are extremely 
sorry for all the confusion and inconvenience that has been caused. We were 
relying on systems that would easily replace and issue SSL Certificates 
automatically, though that didn't occur and we ended up in quite a mess. 
DigiCert didn't work with us to understand the issues and resolve them, we felt 
we were at a dead end.

We'll be following up again shortly with an update surrounding what occurred 
and more infor

RE: How do you handle mass revocation requests?

[Jeremy Rowley via dev-security-policy]

I believe transparency is the best policy.  I think it'd be helpful to the 
community if we could post the email exchange about the revocation. We can 
redact the agreement termination portions if you'd like, but that'd give a lot 
more clarity around what's going on. Do I have your permission to post those 
emails?

-Original Message-
From: dev-security-policy 
 On 
Behalf Of google.manager--- via dev-security-policy
Sent: Wednesday, February 28, 2018 11:54 AM
To: mozilla-dev-security-pol...@lists.mozilla.org
Subject: How do you handle mass revocation requests?

Jeremy,

Today many of our customers experienced lengthy delays when attempting to 
contact us via phone, e-mail and live chat. The reason for the delays were due 
to an unexpected e-mail that DigiCert sent to our customers containing some 
inaccurate information. We were not informed that the e-mail would be sent and 
were caught by surprise. We had disabled this e-mail within your control panel 
and opted to send our own notices, though you took it upon yourselves to send 
my personal details to 20,000 customers.

We didn't authorise DigiCert to contact our customers and we didn't approve the 
content of their e-mail. At no time had any private keys been compromised, nor 
had we ever informed to you that any private keys had been compromised. During 
our many discussions over the past week we put it to you that we believe 
Symantec to have operated our account in a manner whereby it had been 
compromised. Your usage of the word compromise has been twisted by you to your 
benefit and is absolutely defamatory.

We believe the orders placed via our Symantec account were at risk and were 
poorly managed. We have been questioning Symantec without response as to 
concerning items for about a year. Symantec simply ignored our concerns and 
appeared to bury them under the next issue that arose.

There are a range of issues that Trustico intends to investigate via legal 
means.

In good conscience we decided it wasn't ideal to have any active SSL 
Certificates on the Symantec systems, nor any that didn't meet our stringent 
security requirements. Our concerns also relate to the upcoming distrust of all 
Symantec SSL Certificate brands within Google Chrome and the reasoning behind 
it. The same management team responsible for that situation is duly employed at 
DigiCert and are fully managing our account, causing grave concern on our part 
as it appears to be business as usual with a new name. We were also a victim 
whereby Symantec mis-issued SSL Certificates owned by us, subsequently we were 
asked to keep the matter quiet, under a confidentially notice.

We had implemented a system to ensure that all customers would receive a 
replacement SSL Certificate, though today it had failed to perform this 
function.

In our view it is absolutely critical that an SSL Certificate performs its 
intended function. Symantec's issue with Google appeared to seal that deal, 
whereby they will all eventually fail due to distrust. In accordance with CAB 
Forum guidelines we acted to immediately revoke active SSL Certificates whereby 
trust was questionable.

Trustico absolutely distrusts the Symantec brand due to the issues that forced 
Symantec into having to hand over its entire authentication business to an 
alternate CA and a range of issues beforehand. Though, Symantec was ultimately 
acquired by DigiCert - though now DigiCert appears to be influenced by the 
Symantec management team - that to date still is managing our account. Trustico 
stopped offering the Symantec brands early February after a meeting with your 
Symantec management team, whereby they had disclosed to us that various 
reckless issues had occurred (recording available).

We realize that this mass revocation is bothersome and time consuming for all 
that have been affected. We're working to contact all customers to get orders 
replaced as priority and working through a backlog of enquiries. 

Unfortunately, things didn't go very well for us today and we are extremely 
sorry for all the confusion and inconvenience that has been caused. We were 
relying on systems that would easily replace and issue SSL Certificates 
automatically, though that didn't occur and we ended up in quite a mess. 
DigiCert didn't work with us to understand the issues and resolve them, we felt 
we were at a dead end.

We'll be following up again shortly with an update surrounding what occurred 
and more information about where we experienced failures. In the meantime, our 
staff are concentrating on getting SSL Certificates issued as quickly as 
possible from a reputable and trusted CA.

As for the question of who is the subscriber, well ultimately that came down to 
the agreement that we had made with you and the agreement on the website. The 
conditions of the subscriber agreement were not honoured by you when it came to 
our requests anyway 

Re: How do you handle mass revocation requests?

[timx84039--- via dev-security-policy]

We have purchased thousands of certificates using Trustico as a reseller within 
the last years.

Back in these days Trustico created CSR / Private Key pair within their online 
platform (Yes, you read it right - you can create CSR/Private Key on their 
webpage !!!) which was the default at this time and it is still possible to do 
so in their web interface.

Regarding to our investigation they were only able to send the private keys for 
those certificates where the CSR / private key pair were generated within their 
online private key generating tool. This has to be the 23k amount of keys which 
Jeremy received.

I am not aware of guidelines of the CA/B forum but keeping 23.000 (!) private 
keys at your online platform seems more than alarming and is careless and the 
public should be made aware of this fact.

We do not know all aspects of parties involved but we suspect that this could 
be for economic reasons, because Trustico is offering a voucher for Comodo 
certificates in order to replace the revoked RapidSSL/GeoTrust/DigiCerts 
certificates now.
Furthermore they are spreading FUD regarding the upcoming distrust in Google 
Chrome release and trying to force their customers to get a Comodo or 
Trustico-branded certificate.  

Very important: Do not revoke any certificate where Trustico is not able to 
provide the private key!!!
___
dev-security-policy mailing list
dev-security-policy@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-security-policy

Re: How do you handle mass revocation requests?

[Peter Bowen via dev-security-policy]

On Wed, Feb 28, 2018 at 9:37 AM, Jeremy Rowley via dev-security-policy
 wrote:
> Once we were alerted, the team kicked
> off a debate that I wanted to bring to the CAB Forum. Basically, our
> position is that resellers do not constitute subscribers under the Baseline
> Requirement's definitions (Section 1.6.1). As such, we needed to confirm
> that either the key was compromised or that they revocation was authorized
> by the domain holder (the subscriber) prior to revoking the certificate. The
> certificates were not alleged as compromised at that time.

> This raises a question about the MDSP policy and CAB Forum requirements. Who
> is the subscriber in the reseller relation?  We believe this to be the key
> holder. However, the language is unclear. I think we followed the letter and
> spirit of the BRs here, but I'd like feedback, perhaps leading to a ballot
> that clarifies the subscriber in a reseller relationship.

For certs with subject identity information (commonly called IV, OV,
and EV certs), there is no question about the subscriber.  The
Subscriber is the entity identified in the subject: "The Subject is
either the Subscriber or a device under the control and operation of
the Subscriber."

For certificates without subject identity information (DV
certificates), the certificate does not list the subscriber.  However
the CA clearly knows the subscriber, as the subscriber is the "natural
person or Legal Entity to whom a Certificate is issued and who is
legally bound by a Subscriber Agreement or Terms of Use"

In some cases the "reseller" might be the subscriber if the reseller
is a hosting company and is the one that accepts the subscriber
agreement but in the traditional reseller model their customer is the
subscriber as the reseller's customer is the one accepting the
subscriber agreement.

Given that DigiCert appears to have contact information for the
Trustico customers, that suggests that the Trustico customer is likely
the subscriber, but looking at IV/OV/EV certificates (if any) should
tell for sure.

Thanks,
Peter
___
dev-security-policy mailing list
dev-security-policy@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-security-policy

Re: How do you handle mass revocation requests?

[Ryan Sleevi via dev-security-policy]

On Wed, Feb 28, 2018 at 12:37 PM, Jeremy Rowley via dev-security-policy <
dev-security-policy@lists.mozilla.org> wrote:

> On February 2nd, 2018, we received a request from Trustico to mass revoke
> all certificates that had been ordered by end users through Trustico.
> Unfortunately, the email was not sent to the appropriate certificate
> problem
> reporting channels and did not surface immediately so we're delayed in
> sharing the concerns and information. Once we were alerted, the team kicked
> off a debate that I wanted to bring to the CAB Forum. Basically, our
> position is that resellers do not constitute subscribers under the Baseline
> Requirement's definitions (Section 1.6.1). As such, we needed to confirm
> that either the key was compromised or that they revocation was authorized
> by the domain holder (the subscriber) prior to revoking the certificate.
> The
> certificates were not alleged as compromised at that time.
>

I think there's a little nuance to this in the general case (e.g. consider
"Resllers" who are also the hosting provider, and thus constitute the
Applicant/Subscriber even when they're not the domain holder, but
authorized by them), but based on this specific case, I think this sounds
like a correct determination.

I think the biggest question is who agreed to the terms - Trustico or the
end-user - and that mostly impacts the rest of the decision here. Further,
did Trustico warrant on behalf of the user that the user agreed to the
terms (in which case, I would think it's a bit of a copout, and it's really
Trustico agreeing, thus the Subscriber), or did DigiCert have direct
communication with the user on the basis of Trustico's introduction (in
which case, yeah, the Subscriber is the user)

Anyway, just highlighting it here as perhaps not a uniform consensus, but
perhaps not as material as what follows.


> On 2/27/2018, at my request for proof of compromise, we received a file
> with
> 23k private keys matched to specific Trustico customers. This definitely
> triggered our 24-hour revocation processing requirement under 4.9.1.1.3.
> Once we received the keys, we confirmed that these were indeed the matching
> private keys for the reported certificates. We will be revoking these
> certificates today (February 28th, 2018).
>

I think all of this sounds reasonable, no different than a security
researcher (or random member of the public) who were to claim compromise
with no evidence of that.


> Currently, we are only revoking the certificates if we received the private
> keys. There are additional certificates the reseller requested to have
> revoked, but DigiCert has decided to disregard that request until we
> receive
> proof of compromise or more information about the cause of this incident.
>

For the same reason that "Jane Random User" should not be able to cause
revocation merely by assertion, I think that sounds reasonable.


> This raises a question about the MDSP policy and CAB Forum requirements.
> Who
> is the subscriber in the reseller relation?  We believe this to be the key
> holder. However, the language is unclear. I think we followed the letter
> and
> spirit of the BRs here, but I'd like feedback, perhaps leading to a ballot
> that clarifies the subscriber in a reseller relationship.
>

I think the question here is less about who is the Applicant, but who is
the Applicant Representative. If the Reseller is signing/agreeing the
request on behalf of the applicant, or the Subscriber Agreement, they're
the Applicant Representative and ostensibly should be taken as authorized.

I think the gap here is we have this notion of Applicant/Applicant
Representative prior to the issuance - in which some 3P may agree to a
Subscriber Agreement (or warrant agreement), yet claim the Subscriber is
somehow a different entity or that Representative is no longer bound in
scope. That seems pretty troubling - both in terms of how the Applicant
Representative is verified as authorized (which seems fundamentally what a
Reseller who agrees to a ToS is) - and as to how revocation works.


> This also brings up a ballot about the level of due diligence required for
> cert revocation. We generally believe that the private key or demonstration
> of domain control is sufficient to request revocation. Others are at the
> CAs
> discretion. Should we clarify what the due diligence looks like? Are there
> other things we should have done or been doing?
>

I think Wayne's still looking at the revocation space (and I'm much belated
in my offering feedback), but I think one of the gaps is one we've seen
come up with Google domains and Let's Encrypt. And while I share these
stories, to be clear, I don't think LE has done anything wrong in issuance
or handling of it - it just is a good demonstration of the nuance that any
such clarification should consider.

Consider https://crt.sh/?id=245397170 , which was a google.tg certificate
obtained from LE following an apparent Registry compromise. Prior to the

Re: How do you handle mass revocation requests?

[Tom Ritter via dev-security-policy]

On 28 February 2018 at 11:37, Jeremy Rowley via dev-security-policy
 wrote:
> What kind of transparency would the Mozilla community like around this
> issue? There aren't many more facts than I shared above, but there is a lot
> of speculation. Let me know what I can share to help alleviate confusion and
> answer questions.

Have you contacted the customers whose certificates you have not
revoked; but which were in the original batch? It seems likely they're
going to wind up revoked too.

Is there any way to identify these certificates through crt.sh or
through a manual cert search? (Some special
Intermediate/CRL/OID/string...?)

Has Trustico said anything about whether or not they will provide more
information in the future?

-tom
___
dev-security-policy mailing list
dev-security-policy@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-security-policy