Re: [Cryptography-dev] Destroying keys and secrets?

[John Pacific]

Andrew,

If you notice the call to `BN_clear_free`, it zeros the memory of the key
once it gets garbage collected.

However, as several others have explained, this does not prevent the memory
getting stored on the disk in some manner.

-tux

On Feb 22, 2018 14:41, "Andrew W. Donoho"  wrote:

>
>
> > On Feb 20, 2018, at 11:00 , cryptography-dev-requ...@python.org wrote:
> >
> > ec.derive_private_key_from_bytes(secret_bytes, ec.SECP384R1(), backend)
> > could potentially be a way to do this specific operation while reducing
> the
> > number of copies (to zero in Python and 2-3 in OpenSSL, although the
> latter
> > are zeroed), but without tests that can detect non-required copies of
> > secret material it would be extremely hard to prevent regression in the
> > long term as the code is updated.
>
>
>
>
> Paul,
>
>
>
> Based upon your hint above I just went in to the code and looked
> around. It looks like a straightforward extension. Note: I am not a library
> developer and, hence, have not developed all of the various skills to
> properly build pyca/cryptography. Nor am I particularly knowledgable about
> Python v2.7 & v3.6 interoperation issues, much less CPython, Cython and
> PyPy interoperation. The below code is a gedanken exploration to see how
> hard or involved it might actually be to extend pyca/cryptography.
>
> TL;DR: It isn’t hard. I C three functions and then modded them.
> Gratz to the team on an excellent design.
>
> It is clear that I could deploy my own copy of cryptography and
> call it a day. But I believe that key hygiene is an important social good.
> I am happy to help look at/write each routine that imports key material and
> propose versions that allow good key hygiene. As you note above, there is
> no solution for your regression testing issue except to discuss it in code
> commentary. Now to find a routine that can bang the contents of a bytes
> array to 0, 1 and then random.
>
> Should I proceed to engage with your team or just continue on my
> merry way? I want to help but if it isn’t something the team prioritizes
> highly right now, I fully understand.
>
>
>
> Anon,
> Andrew
> 
> Andrew W. Donoho
> Donoho Design Group, L.L.C.
> andrew.don...@gmail.com, +1 (512) 666-7596, twitter.com/adonoho
>
> No risk, no art.
> No art, no reward.
> -- Seth Godin
>
>
>
>
> = backend.py =
>
> def _bytes_to_bn(self, num, bn=None):
> # Was: def _int_to_bn(self, num, bn=None):
> """
> Converts a python bytes array to a BIGNUM. The returned BIGNUM will not
> be garbage collected (to support adding them to structs that take
> ownership of the object). Be sure to register it for GC if it will
> be discarded after use.
> """
> assert bn is None or bn != self._ffi.NULL
>
> if bn is None:
> bn = self._ffi.NULL
>
> bn_ptr = self._lib.BN_bin2bn(num, len(num), bn)
> self.openssl_assert(bn_ptr != self._ffi.NULL)
> return bn_ptr
>
>
> def derive_elliptic_curve_private_key_bytes(self, private_bytes, curve):
> # Was: def derive_elliptic_curve_private_key(self, private_value,
> curve):
> curve_nid = self._elliptic_curve_to_nid(curve)
>
> ec_cdata = self._lib.EC_KEY_new_by_curve_name(curve_nid)
> self.openssl_assert(ec_cdata != self._ffi.NULL)
> ec_cdata = self._ffi.gc(ec_cdata, self._lib.EC_KEY_free)
>
> get_func, group = self._ec_key_determine_group_get_func(ec_cdata)
>
> point = self._lib.EC_POINT_new(group)
> self.openssl_assert(point != self._ffi.NULL)
> point = self._ffi.gc(point, self._lib.EC_POINT_free)
>
> value = self._bytes_to_bn(private_bytes)
> value = self._ffi.gc(value, self._lib.BN_clear_free)
>
> with self._tmp_bn_ctx() as bn_ctx:
> res = self._lib.EC_POINT_mul(group, point, value, self._ffi.NULL,
>  self._ffi.NULL, bn_ctx)
> self.openssl_assert(res == 1)
>
> bn_x = self._lib.BN_CTX_get(bn_ctx)
> bn_y = self._lib.BN_CTX_get(bn_ctx)
>
> res = get_func(group, point, bn_x, bn_y, bn_ctx)
> self.openssl_assert(res == 1)
>
> res = self._lib.EC_KEY_set_public_key(ec_cdata, point)
> self.openssl_assert(res == 1)
> private = self._bytes_to_bn(private_bytes)
> private = self._ffi.gc(private, self._lib.BN_clear_free)
> res = self._lib.EC_KEY_set_private_key(ec_cdata, private)
> self.openssl_assert(res == 1)
>
> evp_pkey = self._ec_cdata_to_evp_pkey(ec_cdata)
>
> return _EllipticCurvePrivateKey(self, ec_cdata, evp_pkey)
>
>
>
> = ec.py =
>
> def derive_private_key_bytes(private_value, curve, backend):
> # Was: def derive_private_key(private_value, curve, backend):
>if not isinstance(curve, EllipticCurve):
> raise TypeError("curve must provide the EllipticCurve interface.")
>
> return 

Re: [Cryptography-dev] Destroying keys and secrets?

[Andrew W. Donoho]

> On Feb 20, 2018, at 11:00 , cryptography-dev-requ...@python.org wrote:
> 
> ec.derive_private_key_from_bytes(secret_bytes, ec.SECP384R1(), backend)
> could potentially be a way to do this specific operation while reducing the
> number of copies (to zero in Python and 2-3 in OpenSSL, although the latter
> are zeroed), but without tests that can detect non-required copies of
> secret material it would be extremely hard to prevent regression in the
> long term as the code is updated.




Paul,



Based upon your hint above I just went in to the code and looked 
around. It looks like a straightforward extension. Note: I am not a library 
developer and, hence, have not developed all of the various skills to properly 
build pyca/cryptography. Nor am I particularly knowledgable about Python v2.7 & 
v3.6 interoperation issues, much less CPython, Cython and PyPy interoperation. 
The below code is a gedanken exploration to see how hard or involved it might 
actually be to extend pyca/cryptography. 

TL;DR: It isn’t hard. I C three functions and then modded them. Gratz 
to the team on an excellent design.

It is clear that I could deploy my own copy of cryptography and call it 
a day. But I believe that key hygiene is an important social good. I am happy 
to help look at/write each routine that imports key material and propose 
versions that allow good key hygiene. As you note above, there is no solution 
for your regression testing issue except to discuss it in code commentary. Now 
to find a routine that can bang the contents of a bytes array to 0, 1 and then 
random.

Should I proceed to engage with your team or just continue on my merry 
way? I want to help but if it isn’t something the team prioritizes highly right 
now, I fully understand.



Anon,
Andrew

Andrew W. Donoho
Donoho Design Group, L.L.C.
andrew.don...@gmail.com, +1 (512) 666-7596, twitter.com/adonoho

No risk, no art.
No art, no reward.
-- Seth Godin




= backend.py =

def _bytes_to_bn(self, num, bn=None):
# Was: def _int_to_bn(self, num, bn=None):
"""
Converts a python bytes array to a BIGNUM. The returned BIGNUM will not
be garbage collected (to support adding them to structs that take
ownership of the object). Be sure to register it for GC if it will
be discarded after use.
"""
assert bn is None or bn != self._ffi.NULL

if bn is None:
bn = self._ffi.NULL

bn_ptr = self._lib.BN_bin2bn(num, len(num), bn)
self.openssl_assert(bn_ptr != self._ffi.NULL)
return bn_ptr


def derive_elliptic_curve_private_key_bytes(self, private_bytes, curve):
# Was: def derive_elliptic_curve_private_key(self, private_value, curve):
curve_nid = self._elliptic_curve_to_nid(curve)

ec_cdata = self._lib.EC_KEY_new_by_curve_name(curve_nid)
self.openssl_assert(ec_cdata != self._ffi.NULL)
ec_cdata = self._ffi.gc(ec_cdata, self._lib.EC_KEY_free)

get_func, group = self._ec_key_determine_group_get_func(ec_cdata)

point = self._lib.EC_POINT_new(group)
self.openssl_assert(point != self._ffi.NULL)
point = self._ffi.gc(point, self._lib.EC_POINT_free)

value = self._bytes_to_bn(private_bytes)
value = self._ffi.gc(value, self._lib.BN_clear_free)

with self._tmp_bn_ctx() as bn_ctx:
res = self._lib.EC_POINT_mul(group, point, value, self._ffi.NULL,
 self._ffi.NULL, bn_ctx)
self.openssl_assert(res == 1)

bn_x = self._lib.BN_CTX_get(bn_ctx)
bn_y = self._lib.BN_CTX_get(bn_ctx)

res = get_func(group, point, bn_x, bn_y, bn_ctx)
self.openssl_assert(res == 1)

res = self._lib.EC_KEY_set_public_key(ec_cdata, point)
self.openssl_assert(res == 1)
private = self._bytes_to_bn(private_bytes)
private = self._ffi.gc(private, self._lib.BN_clear_free)
res = self._lib.EC_KEY_set_private_key(ec_cdata, private)
self.openssl_assert(res == 1)

evp_pkey = self._ec_cdata_to_evp_pkey(ec_cdata)

return _EllipticCurvePrivateKey(self, ec_cdata, evp_pkey)



= ec.py =

def derive_private_key_bytes(private_value, curve, backend):
# Was: def derive_private_key(private_value, curve, backend):
   if not isinstance(curve, EllipticCurve):
raise TypeError("curve must provide the EllipticCurve interface.")

return backend.derive_elliptic_curve_private_key_bytes(private_value, curve)




___
Cryptography-dev mailing list
Cryptography-dev@python.org
https://mail.python.org/mailman/listinfo/cryptography-dev

Re: [Cryptography-dev] Destroying keys and secrets?

[Andrew W. Donoho]

> On Feb 20, 2018, at 11:00 , cryptography-dev-requ...@python.org wrote:
> Date: Mon, 19 Feb 2018 17:14:25 -0800
> From: Paul Kehrer 
> To: cryptography-dev@python.org
> Subject: Re: [Cryptography-dev] Cryptography-dev Digest, Vol 54, Issue
>   2
> Message-ID:
>   
> Content-Type: text/plain; charset="utf-8”
> 
> Access to Python's memory (via side channel or dumping as root) is not part
> of pyca/cryptography's threat model at this time so we don't attempt to
> protect against it. Making it part of our threat model would be difficult
> due in part to the reasons you stated above as well as the difficulty in
> writing tests to prevent regression, but let's talk about what CPython does
> in this case.



Paul,



Thank you for your enlightening comments above and below.



What Am I Really Trying To Do?:

This function is initially an AWS lambda function that I also intend to support 
on Google Cloud and MS Azure. This august group of engineers obviously knows of 
my concern but I nonetheless wish to emphasize the problem. Perhaps I can also 
contribute to its solution? While I have control of the function’s environment, 
I don’t have total control. For example, while I can hope the host OS has 
enough entropy in its pool to support os.urandom(), the AWS recommendation, 
with which I concur, is that I get my random_bytes from the AWS KMS service and 
stir it up with an HKDF. As in:



random_bytes = os.urandom(512//8)  # <= Should come from AWS KMS random bytes 
service.

# Derive a SECP384R1 private key using SHA-512, _salt512, and guid.
hkdf = HKDF(algorithm=sha512, length=384//8, salt=_salt512, info=guid, 
backend=backend)
secret_bytes = hkdf.derive(random_bytes)
secret_int = int.from_bytes(secret_bytes, byteorder='big')
private_key = ec.derive_private_key(secret_int, ec.SECP384R1(), backend)



As I understand it, each of the cloud vendors offers a high entropy source of 
random bytes. Upon completion of this function, I want to scrub RAM of key 
material. I will never get another chance to address the existence of this key 
material. There is no goodbye kiss from the cloud function execution 
environments. These are one-shot function calls. If I am to ensure that this 
crypto toxic waste doesn’t come back to bite my service, then I must dispose of 
it in the function context that created it. 



In other words, pyca/cryptography is likely to be used in a much more dynamic 
environment than heretofore. Considering that pyca/cryptography has largely 
succeeded in building a civilized interface to crypto routines, helping folks 
implement good secret/key hygiene seems in scope for the project’s goals. 
Considering Amazon’s embrace of pyca/cryptography for their Python lambda 
functions and AWS Encryption SDK, I am unlikely to be the sole user.



> int.from_bytes will unfortunately make a copy (to a Python integer). That
> int will then be copied into a BN via _int_to_bn (
> https://github.com/pyca/cryptography/blob/master/src/cryptography/hazmat/backends/openssl/backend.py#L317-L346)
> when you call derive_private_key. It will actually be converted twice (a
> thing we should fix) (
> https://github.com/pyca/cryptography/blob/master/src/cryptography/hazmat/backends/openssl/backend.py#L1383-L1419).
> Although the resulting BNs will themselves be zeroed as freed, this means a
> secret scalar bytestring created in Python will be resident in memory no
> less than 5 times (3 byte strings, 2 numbers).
> 
> Obviously the next logical question is why you'd provide a Python integer
> when we're just going to convert it back to big endian bytes anyway.
> Disregarding the memory clearing issue it's also inefficient. When
> originally designing some of the APIs we made a mistake and chose integers
> instead of big endian bytes (see: numbers classes). We have not yet added
> alternate APIs to potentially enable us to deprecate numbers because the
> improvement in efficiency probably isn't worth the pain of trying to
> convert the huge number of users of those classes.
> 
> ec.derive_private_key_from_bytes(secret_bytes, ec.SECP384R1(), backend)
> could potentially be a way to do this specific operation while reducing the
> number of copies (to zero in Python and 2-3 in OpenSSL, although the latter
> are zeroed), but without tests that can detect non-required copies of
> secret material it would be extremely hard to prevent regression in the
> long term as the code is updated.



Thank you for the above excellent exposition of the state of data 
copying in pyca/cryptography. Considering the appropriate "stürm und drang" 
that accompanied Meltdown, Specter, and other exploits around key material, I 
think many developers have an interest in making the changes to their code that 
allows them to ensure precise key material lifetimes. Of course, with modern 
internet survey tools, the pyca/cryptography