Re: Cert hot-reloading

[Kyle Hamilton]

Could this be dealt with by the simple removal of any caching layer
between an SSL_CTX and a directory processed by openssl c_rehash?
Would reading the filesystem on every certificate verification be too
heavy for your use case?

On Sun, Aug 30, 2020 at 7:20 PM Jordan Brown
 wrote:
>
> Well, I can restate the problem that I encountered.
>
> We deliver an integrated storage system.  Under the covers it is a modified 
> Solaris running a usual collection of proprietary and open-source components. 
>  We supply an administrative user interface that, among many other things, 
> lets you manage a list of "trusted" certificates - typically CA certificates 
> that a program would use to authenticate its peers.  That is, it's the 
> equivalent of Firefox's Tools / Options / Privacy & Security / Certificates / 
> View Certificates, and the "Servers" and "Authorities" tabs there, with the 
> additional tidbit that for each certificate you can control which services 
> (e.g. LDAP, et cetera) that certificate is trusted for.
>
> When an administrator makes a change to the trusted-certificates list, we 
> want that change to take effect, system-wide.
>
> The problem is that that means that some number of processes with active 
> OpenSSL contexts need to drop those contexts and recreate them, and we don't 
> know which processes those are.  Client operations are typically driven 
> through a library, not a separate daemon, and so there's no centralized way 
> to know which processes might be TLS clients.  In addition, there's the 
> question of how to *tell* the process to recreate the context.  Simply 
> restarting them may involve disruption of various sorts.
>
> What we'd like would be for OpenSSL to, on every authentication, stat the 
> file or directory involved, and if it's changed then wipe the in-memory cache.
>
> Yes, aspects of this are system-specific, but that's true of many things.  
> There could easily be an internal API that captures a current-stage object, 
> and another that answers "is this still the same".  The default 
> implementation could always say "yes".
>
> --
> Jordan Brown, Oracle ZFS Storage Appliance, Oracle Solaris

Re: Cert hot-reloading

[David Arnold]

Should aspects of an implementation be configurable behavior with a 
sane default? I'd guess so...


Hot-plugging the pointer seems to force atomicity considerations 
down-stream, which might be
educationally a good thing for openssl to press for. It also addresses 
Jordan's use case, for however
application specific it might be. For compat reasons, a "legacy" mode 
which creates a new context
for *new* connections might be the necessary "bridge" into that 
transformation.


For change detection: I think "on next authentication" has enough (or 
even better) guarantees over a periodic loop.


For file read atomicity: What are the options to keep letsencrypt & co 
at comfort? Although the hereditary
"right (expectation) for comfort" is somewhat offset by a huge gain in 
functionality. It still feels like a convincing deal.


- add a staleness check on every change detection? (maybe costly?)
- consume a tar if clients want those guarantees? (opt-out or opt-out?)




On Sun, Aug 30, 2020 at 19:54, Kyle Hamilton  wrote:
I'm not sure I can follow the "in all cases it's important to keep 
the key and cert in the same file" argument, particularly in line 
with openat() usage on the cert file after privilege to open the key 
file has been dropped.  I agree that key/cert staleness is important 
to address in some manner, but I don't think it's necessarily 
appropriate here.


I also don't think it's necessarily okay to add a new requirement 
that e.g. letsencrypt clients reconcatentate their keys and certs, 
and that all of the Apache-style configuration guides be rewritten to 
consolidate the key and cert files. On a simple certificate renewal 
without a rekey, the best current practice is sufficient.  (As well, 
a letsencrypt client would possibly need to run privileged in that 
scenario to reread the private key file in order to reconcatenate it, 
which is not currently actually necessary.  Increasing the privileges 
required for any non-OS service for any purpose that isn't related to 
OS kernel privilege requirements feels a bit disingenuous.)


Of course, if you want to alter the conditions which led to the best 
current practice (and impose retraining on everyone), that's a 
different matter.  But I still think increasing privilege 
requirements would be a bad thing, under the least-privilege 
principle.


-Kyle H

On Sun, Aug 30, 2020, 18:36 Viktor Dukhovni 
mailto:openssl-us...@dukhovni.org>> 
wrote:

On Sun, Aug 30, 2020 at 05:45:41PM -0500, David Arnold wrote:

 > If you prefer this mailing list over github issues, I still want 
to ask

 > for comments on:
 >
 > Certificate hot-reloading #12753
 > <>
 >
 > Specifically, my impression is that this topic has died down a 
bit and
 > from the linked mailing list threads, in my eye, no concrete 
conclusion

 > was drawn.
 >
 > I'm not sure how to rank this motion in the context of OpenSSL
 > development, but I guess OpenSSL is used to producing ripple 
effects,

 > so the man-hour argument might be a genuinely valid one.
 >
 > Please inform my research about this issue with your comments!

 This is a worthwhile topic.  It has a few interesting aspects:

 1.  Automatic key+cert reloads upon updates of key+cert chain 
PEM

 files.  This can be tricky when processes start privileged,
 load the certs and then drop privs, and are no longer able
 to reopen the key + cert chain file.

 - Here, for POSIX systems I'd go with an approach where
   it is the containing directory that is restricted to
   root or similar, and the actual cert files are group
   and or world readable.  The process can then keep
   the directory file descriptor open, and then openat(2)
   to periodically check the cert file, reloading when
   the metadata changes.

 - With non-POSIX systems, or applications that don't
   drop privs, the openat(2) is not needed, and one
   just checks the cert chain periodically.

 - Another option is to use passphrase-protected keys,
   and load the secret passphrase at process start from
   a separate read-protected file, while the actual
   private key + cert chain file is world readable,
   with the access control via protecting the passphrase
   file.

 - In all cases, it is important to keep both the private
   key and the cert in the same file, and open it just
   once to read both, avoiding races in which the key
   and cert are read in a way that results in one or
   the other being stale.

 2.  Having somehow obtained a new key + cert chain, one
 now wants to non-disruptively apply them to running
 servers.  Here there are two potential approaches:

 - Hot plug a new pointer into an existing SSL_CTX structure.
   While the update itself could be made atomic, the readers
 

Re: Cert hot-reloading

[Kyle Hamilton]

I'm not sure I can follow the "in all cases it's important to keep the key
and cert in the same file" argument, particularly in line with openat()
usage on the cert file after privilege to open the key file has been
dropped.  I agree that key/cert staleness is important to address in some
manner, but I don't think it's necessarily appropriate here.

I also don't think it's necessarily okay to add a new requirement that e.g.
letsencrypt clients reconcatentate their keys and certs, and that all of
the Apache-style configuration guides be rewritten to consolidate the key
and cert files. On a simple certificate renewal without a rekey, the best
current practice is sufficient.  (As well, a letsencrypt client would
possibly need to run privileged in that scenario to reread the private key
file in order to reconcatenate it, which is not currently actually
necessary.  Increasing the privileges required for any non-OS service for
any purpose that isn't related to OS kernel privilege requirements feels a
bit disingenuous.)

Of course, if you want to alter the conditions which led to the best
current practice (and impose retraining on everyone), that's a different
matter.  But I still think increasing privilege requirements would be a bad
thing, under the least-privilege principle.

-Kyle H

On Sun, Aug 30, 2020, 18:36 Viktor Dukhovni 
wrote:

> On Sun, Aug 30, 2020 at 05:45:41PM -0500, David Arnold wrote:
>
> > If you prefer this mailing list over github issues, I still want to ask
> > for comments on:
> >
> > Certificate hot-reloading #12753
> > 
> >
> > Specifically, my impression is that this topic has died down a bit and
> > from the linked mailing list threads, in my eye, no concrete conclusion
> > was drawn.
> >
> > I'm not sure how to rank this motion in the context of OpenSSL
> > development, but I guess OpenSSL is used to producing ripple effects,
> > so the man-hour argument might be a genuinely valid one.
> >
> > Please inform my research about this issue with your comments!
>
> This is a worthwhile topic.  It has a few interesting aspects:
>
> 1.  Automatic key+cert reloads upon updates of key+cert chain PEM
> files.  This can be tricky when processes start privileged,
> load the certs and then drop privs, and are no longer able
> to reopen the key + cert chain file.
>
> - Here, for POSIX systems I'd go with an approach where
>   it is the containing directory that is restricted to
>   root or similar, and the actual cert files are group
>   and or world readable.  The process can then keep
>   the directory file descriptor open, and then openat(2)
>   to periodically check the cert file, reloading when
>   the metadata changes.
>
> - With non-POSIX systems, or applications that don't
>   drop privs, the openat(2) is not needed, and one
>   just checks the cert chain periodically.
>
> - Another option is to use passphrase-protected keys,
>   and load the secret passphrase at process start from
>   a separate read-protected file, while the actual
>   private key + cert chain file is world readable,
>   with the access control via protecting the passphrase
>   file.
>
> - In all cases, it is important to keep both the private
>   key and the cert in the same file, and open it just
>   once to read both, avoiding races in which the key
>   and cert are read in a way that results in one or
>   the other being stale.
>
> 2.  Having somehow obtained a new key + cert chain, one
> now wants to non-disruptively apply them to running
> servers.  Here there are two potential approaches:
>
> - Hot plug a new pointer into an existing SSL_CTX structure.
>   While the update itself could be made atomic, the readers
>   of such pointers might read them more than once to separately
>   extract the key and the cert chain, without checking that
>   they're using the same pointer for both operations.
>
>   This is bound to be fragile, though not necessarily
>   impossible.
>
> - Build a new SSL_CTX, and use it to accept *new* connections,
>   while existing connections use whatever SSL_CTX they started
>   with.  I believe this can work well, because "SSL" handles
>   increment the reference count of the associated SSL_CTX
>   when they're created, and decrement it when destroyed.
>
>   So when you create a replacement SSL_CTX, you can just
>   SSL_CTX_free() the old, and it will only actually
>   be deleted when the last SSL connection tied to that
>   SSL_CTX is destroyed.
>
>   It is true that typical SSL_CTX construction is modestly
>   expensive (loading CA stores and the like) but some of
>   that could be 

Re: Cert hot-reloading

[Jordan Brown]

Well, I can restate the problem that I encountered.

We deliver an integrated storage system.  Under the covers it is a
modified Solaris running a usual collection of proprietary and
open-source components.  We supply an administrative user interface
that, among many other things, lets you manage a list of "trusted"
certificates - typically CA certificates that a program would use to
authenticate its peers.  That is, it's the equivalent of Firefox's Tools
/ Options / Privacy & Security / Certificates / View Certificates, and
the "Servers" and "Authorities" tabs there, with the additional tidbit
that for each certificate you can control which services (e.g. LDAP, et
cetera) that certificate is trusted for.

When an administrator makes a change to the trusted-certificates list,
we want that change to take effect, system-wide.

The problem is that that means that some number of processes with active
OpenSSL contexts need to drop those contexts and recreate them, and we
don't know which processes those are.  Client operations are typically
driven through a library, not a separate daemon, and so there's no
centralized way to know which processes might be TLS clients.  In
addition, there's the question of how to *tell* the process to recreate
the context.  Simply restarting them may involve disruption of various
sorts.

What we'd like would be for OpenSSL to, on every authentication, stat
the file or directory involved, and if it's changed then wipe the
in-memory cache.

Yes, aspects of this are system-specific, but that's true of many
things.  There could easily be an internal API that captures a
current-stage object, and another that answers "is this still the
same".  The default implementation could always say "yes".

-- 
Jordan Brown, Oracle ZFS Storage Appliance, Oracle Solaris

Re: Cert hot-reloading

[Karl Denninger]

On 8/30/2020 19:28, Viktor Dukhovni wrote:

On Sun, Aug 30, 2020 at 05:45:41PM -0500, David Arnold wrote:


If you prefer this mailing list over github issues, I still want to ask
for comments on:

Certificate hot-reloading #12753


Specifically, my impression is that this topic has died down a bit and
from the linked mailing list threads, in my eye, no concrete conclusion
was drawn.

I'm not sure how to rank this motion in the context of OpenSSL
development, but I guess OpenSSL is used to producing ripple effects,
so the man-hour argument might be a genuinely valid one.

Please inform my research about this issue with your comments!

This is a worthwhile topic.  It has a few interesting aspects:

 1.  Automatic key+cert reloads upon updates of key+cert chain PEM
 files.  This can be tricky when processes start privileged,
 load the certs and then drop privs, and are no longer able
 to reopen the key + cert chain file.

 - Here, for POSIX systems I'd go with an approach where
   it is the containing directory that is restricted to
   root or similar, and the actual cert files are group
   and or world readable.  The process can then keep
   the directory file descriptor open, and then openat(2)
   to periodically check the cert file, reloading when
   the metadata changes.

 - With non-POSIX systems, or applications that don't
   drop privs, the openat(2) is not needed, and one
   just checks the cert chain periodically.

 - Another option is to use passphrase-protected keys,
   and load the secret passphrase at process start from
   a separate read-protected file, while the actual
   private key + cert chain file is world readable,
   with the access control via protecting the passphrase
   file.

 - In all cases, it is important to keep both the private
   key and the cert in the same file, and open it just
   once to read both, avoiding races in which the key
   and cert are read in a way that results in one or
   the other being stale.

 2.  Having somehow obtained a new key + cert chain, one
 now wants to non-disruptively apply them to running
 servers.  Here there are two potential approaches:

 - Hot plug a new pointer into an existing SSL_CTX structure.
   While the update itself could be made atomic, the readers
   of such pointers might read them more than once to separately
   extract the key and the cert chain, without checking that
   they're using the same pointer for both operations.

   This is bound to be fragile, though not necessarily
   impossible.

 - Build a new SSL_CTX, and use it to accept *new* connections,
   while existing connections use whatever SSL_CTX they started
   with.  I believe this can work well, because "SSL" handles
   increment the reference count of the associated SSL_CTX
   when they're created, and decrement it when destroyed.

   So when you create a replacement SSL_CTX, you can just
   SSL_CTX_free() the old, and it will only actually
   be deleted when the last SSL connection tied to that
   SSL_CTX is destroyed.

   It is true that typical SSL_CTX construction is modestly
   expensive (loading CA stores and the like) but some of
   that could be handled by sharing and reference-counting
   the stores.

So my preferred approach would be to create a new SSL_CTX, and get new
connections using that.  Now in a multi-threaded server, it could be a
bit tricky to ensure that the SSL_CTX_free() does not happen before all
threads reading the pointer to the latest SSL_CTX see the new pointer
installed.  Something equivalent to RCU may be needed to ensure that the
free only happens after the new pointer is visible in all threads.

Designs addressing various parts of this would be cool, provided they're
well thought out, and not just single-use-case quick hacks.


This works now; I use it with an application that checks in with a 
license server and can grab a new cert.  OpenSSL appears to have no 
problem with setting up a new SSL_CTX and using it for new connections; 
the old ones continue onward until they terminate, and new ones are fine 
as well.


This appears to be be ok with the current code; I've yet to have it blow 
up in my face although at present the certs in question are reasonably 
long-lived.  Whether it's robust enough to handle very short-term 
certificates I do not know.


--
Karl Denninger
k...@denninger.net 
/The Market Ticker/
/[S/MIME encrypted email preferred]/


smime.p7s
Description: S/MIME Cryptographic Signature

Re: Cert hot-reloading

[Viktor Dukhovni]

On Sun, Aug 30, 2020 at 05:45:41PM -0500, David Arnold wrote:

> If you prefer this mailing list over github issues, I still want to ask 
> for comments on:
> 
> Certificate hot-reloading #12753
> 
> 
> Specifically, my impression is that this topic has died down a bit and 
> from the linked mailing list threads, in my eye, no concrete conclusion 
> was drawn.
> 
> I'm not sure how to rank this motion in the context of OpenSSL
> development, but I guess OpenSSL is used to producing ripple effects,
> so the man-hour argument might be a genuinely valid one.
> 
> Please inform my research about this issue with your comments!

This is a worthwhile topic.  It has a few interesting aspects:

1.  Automatic key+cert reloads upon updates of key+cert chain PEM
files.  This can be tricky when processes start privileged,
load the certs and then drop privs, and are no longer able
to reopen the key + cert chain file.

- Here, for POSIX systems I'd go with an approach where
  it is the containing directory that is restricted to
  root or similar, and the actual cert files are group
  and or world readable.  The process can then keep
  the directory file descriptor open, and then openat(2)
  to periodically check the cert file, reloading when
  the metadata changes.

- With non-POSIX systems, or applications that don't
  drop privs, the openat(2) is not needed, and one
  just checks the cert chain periodically.

- Another option is to use passphrase-protected keys,
  and load the secret passphrase at process start from
  a separate read-protected file, while the actual
  private key + cert chain file is world readable,
  with the access control via protecting the passphrase
  file.

- In all cases, it is important to keep both the private
  key and the cert in the same file, and open it just
  once to read both, avoiding races in which the key
  and cert are read in a way that results in one or
  the other being stale.

2.  Having somehow obtained a new key + cert chain, one
now wants to non-disruptively apply them to running
servers.  Here there are two potential approaches:

- Hot plug a new pointer into an existing SSL_CTX structure.
  While the update itself could be made atomic, the readers
  of such pointers might read them more than once to separately
  extract the key and the cert chain, without checking that
  they're using the same pointer for both operations.

  This is bound to be fragile, though not necessarily
  impossible.

- Build a new SSL_CTX, and use it to accept *new* connections,
  while existing connections use whatever SSL_CTX they started
  with.  I believe this can work well, because "SSL" handles
  increment the reference count of the associated SSL_CTX
  when they're created, and decrement it when destroyed.

  So when you create a replacement SSL_CTX, you can just
  SSL_CTX_free() the old, and it will only actually
  be deleted when the last SSL connection tied to that
  SSL_CTX is destroyed.

  It is true that typical SSL_CTX construction is modestly
  expensive (loading CA stores and the like) but some of
  that could be handled by sharing and reference-counting
  the stores.

So my preferred approach would be to create a new SSL_CTX, and get new
connections using that.  Now in a multi-threaded server, it could be a
bit tricky to ensure that the SSL_CTX_free() does not happen before all
threads reading the pointer to the latest SSL_CTX see the new pointer
installed.  Something equivalent to RCU may be needed to ensure that the
free only happens after the new pointer is visible in all threads.

Designs addressing various parts of this would be cool, provided they're
well thought out, and not just single-use-case quick hacks.

-- 
Viktor.

Cert hot-reloading

[David Arnold]

Hi,

If you prefer this mailing list over github issues, I still want to ask 
for comments on:


Certificate hot-reloading #12753


Specifically, my impression is that this topic has died down a bit and 
from the linked mailing list threads, in my eye, no concrete conclusion 
was drawn.


I'm not sure how to rank this motion in the context of OpenSSL 
development, but I guess OpenSSL is used to producing ripple effects, 
so the man-hour

argument might be a genuinely valid one.

Please inform my research about this issue with your comments!

BR, David A

ASN1 integer conversion - why is this correct ?

[Dirk-Willem van Gulik]

I am converting an unsigned integer (P,Q of an ECDSA 256 bit curve) from a 32 
byte array (as provided by Microsoft its .NET cryptographic framework) to an 
ANS1_INTEGER.

The steps taken are:

unsigned char in[32] = ..

r = BN_bin2bn(in, 32, NULL); 
BN_to_ASN1_INTEGER(r, asn1intptr);

All works well; except for these two test cases:

in[]32 = 
FF F0 00 00 00 00 00 00   00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00   00 00 00 00 00 00 00 00

in[]32 = 
FF F0 00 00 00 00 00 00   00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00   00 00 00 00 00 00 00 FF // < last 
bits set

Which both yield:

 2:d=1  hl=2 l=  33 prim: INTEGER   
:EBFFF000

And

in[]32 = 03 00 00 00 00 00 00 00   00 00 00 00 00 00 00 00
 00 00 00 00 00 00 00 00   00 00 00 00 00 00 00 FF 

Which yields:

 37:d=1  hl=2 l=  33 prim: INTEGER   
:FF03

Could someone explain me what happens here, especially to the last 0xFF bits ?

With kind regards,

Actual code at [1]; test script output of gen-tc.sh[2] in [3].

Dw.

1: 
https://github.com/minvws/nl-contact-tracing-odds-and-ends/tree/master/dotNet_ms64_to_x962
2: 
https://github.com/minvws/nl-contact-tracing-odds-and-ends/blob/master/dotNet_ms64_to_x962/gen-tc.sh
3: 
https://github.com/minvws/nl-contact-tracing-odds-and-ends/blob/master/dotNet_ms64_to_x962/test.txt