Re: [openssl-dev] Plea for a new public OpenSSL RNG API

[Blumenthal, Uri - 0553 - MITLL]

I like this PR. Thank you!

> On Sep 3, 2017, at 17:53, Dr. Matthias St. Pierre 
>  wrote:
> 
>> 
>> The 'RAND_add()/RAND_bytes()' pattern is broken
>> ===
>> 
>> In OpenSSL, the classical way for the RNG consumer to add his own randomness 
>> is to call 'RAND_add()' before
>> calling 'RAND_bytes()'. If the new 'RAND_OpenSSL()' method (the 
>> "compatibility layer" hiding the public
>> RAND_DRBG instance)  is the default, then this does not work as expected 
>> anymore:
>> 
>> The reason is that a call to 'RAND_add()' adds the provided randomness only 
>> to a global buffer
>> ('rand_bytes'), from which it will be pulled during the next reseed. But no 
>> reseed is triggered. So the next
>> RAND_bytes() call will be unaffected from the RAND_add(), which is not what 
>> the consumer expected. (The same
>> holds for 'RAND_seed()', since 'drbg_seed()' only calls into 'drbg_add()')
>> 
>> Reseeding of DRBGs occurs only at the following occasions:
>> 
>> * immediately after a 'fork()' (new)
>> * if the 'reseed_counter' exceeds the 'reseed_interval'
>> * if 'RAND_DRBG_generate()' is called requesting 'prediction_resistance'
>> * 'RAND_DRBG_reseed()' is called explicitely
>> 
>> *Note:* Currently it looks like the situation is even worse: if 'RAND_add()' 
>> is called multiple times before
>> a reseed occurs, then the result of the previous call is overwritten.
> 
> 
> I just posted GitHub PR #4328 related to this issue
> 
>   [openssl/openssl] WIP: Fix the RAND_add() reseeding issue (#4328)
> 
> 
> Matthias St. Pierre
-- 
openssl-dev mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev

Re: [openssl-dev] Plea for a new public OpenSSL RNG API

[Dr. Matthias St. Pierre]

>
> The 'RAND_add()/RAND_bytes()' pattern is broken
> ===
> 
> In OpenSSL, the classical way for the RNG consumer to add his own randomness 
> is to call 'RAND_add()' before
> calling 'RAND_bytes()'. If the new 'RAND_OpenSSL()' method (the 
> "compatibility layer" hiding the public
> RAND_DRBG instance)  is the default, then this does not work as expected 
> anymore:
> 
> The reason is that a call to 'RAND_add()' adds the provided randomness only 
> to a global buffer
> ('rand_bytes'), from which it will be pulled during the next reseed. But no 
> reseed is triggered. So the next
> RAND_bytes() call will be unaffected from the RAND_add(), which is not what 
> the consumer expected. (The same
> holds for 'RAND_seed()', since 'drbg_seed()' only calls into 'drbg_add()')
> 
> Reseeding of DRBGs occurs only at the following occasions:
> 
> * immediately after a 'fork()' (new)
> * if the 'reseed_counter' exceeds the 'reseed_interval'
> * if 'RAND_DRBG_generate()' is called requesting 'prediction_resistance'
> * 'RAND_DRBG_reseed()' is called explicitely
> 
> *Note:* Currently it looks like the situation is even worse: if 'RAND_add()' 
> is called multiple times before
> a reseed occurs, then the result of the previous call is overwritten.


I just posted GitHub PR #4328 related to this issue

[openssl/openssl] WIP: Fix the RAND_add() reseeding issue (#4328)

 
Matthias St. Pierre



smime.p7s
Description: S/MIME cryptographic signature
-- 
openssl-dev mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev

Re: [openssl-dev] Plea for a new public OpenSSL RNG API

[Paul Dale]

To access a PKCS#11 randomness source, it would be necessary to have an engine 
that implemented whatever new RNG API is defined which in turn talks to the P11 
device.  Possibly not ideal but workable.

As for the entropy argument to RAND_add et al, the callee will use it in a 
manner suitable to it.  For a DRBG, the buffer will likely be added as 
additional data and the argument ignored.  Other RNGs could use it differently. 
 NIST 800-90B 3.1.5.1.2 specifies the minimum amount of entropy that must be 
input in order achieve a desired output.  In this case the RNG would accumulate 
the entropy arguments until it achieves the requisite level.

The trust issue is both harder and easier.  I agree completely that working out 
if you trust the assessed entropy or not is an incredibly difficult 
(impossible) task.  However, we're a library and the user is telling us what 
_they_ trust.  It isn't OpenSSL's place to try to second guess this, issue a 
warning at best.  We don't stop users choosing poor passwords, using zero as an 
IV or generating a sixteen bit RSA key.


As for the new RNG engine API, I've been considering the benefits of having two 
calls: one to get random bytes, the other to request entropy.  The first can be 
whitened or produced by a DRBG etc, the second also returns an estimate as to 
the quality.  Essentially the difference between RDRAND and RDSEED.


Pauli

-- 
Oracle
Dr Paul Dale | Cryptographer | Network Security & Encryption 
Phone +61 7 3031 7217
Oracle Australia

-Original Message-
From: Blumenthal, Uri - 0553 - MITLL [mailto:u...@ll.mit.edu] 
Sent: Thursday, 31 August 2017 12:27 AM
To: openssl-dev@openssl.org
Subject: Re: [openssl-dev] Plea for a new public OpenSSL RNG API

On 8/30/17, 00:59, "openssl-dev on behalf of Paul Dale" 
<openssl-dev-boun...@openssl.org on behalf of paul.d...@oracle.com> wrote:

>My thoughts are that the new RNG API should be made public once it has
>been properly designed.  We've a chance to get this right, let's take the 
> time
>and make an effort to do so.  There is no rush.

Not quite. If there is an RNG involved in generating long-term keys now – users 
better be able to control/affect it now.

>   I also feel that having an engine interface to the new RNG API is 
> worthwhile.

+1

>   It allows hardware sources to be used via the same API.

I rather doubt this. For example, my smartcard (accessible via PKCS#11) is a 
hardware source, which I occasionally use. How do you see it used with the same 
API?

>I would like to see an entropy argument to the equivalent to RAND_add.
>   Anyone is welcome to always pass zero in but there should be the option to 
> not. 
>   Consider an hardware source with _provable_ output quality, why shouldn't 
> it be
>   allowed to meaningfully contribute?

What’s the purpose of passing the entropy argument? How is the callee (the RNG) 
going to use it? Why should the OpenSSL code in general trust the received 
value (how can OpenSSL tell that the received randomness is indeed from a 
hardware source with provable output quality)? Finally, what does it matter? 

And they all “meaningfully contribute”. The only question is whether this 
contribution should prevent RNG from acquiring more entropy from other sources. 
My opinion is resounding no.

>   I like the idea of two independent global RNGs.

+1  ;-)

>  This does increase seeding requirements however.

If you can seed one, you can seed two.



-Original Message-
From: Dr. Matthias St. Pierre [mailto:matthias.st.pie...@ncp-e.com] 
Sent: Tuesday, 29 August 2017 7:45 PM
To: openssl-dev@openssl.org
Subject: [openssl-dev] Plea for a new public OpenSSL RNG API

Hi everybody,

on the [openssl-dev] mailing list, there has been a long ongoing discussion 
about the new RAND_DRBG API and comparing it with the old RAND_METHOD API (see 
"[openssl-dev] Work on a new RNG for OpenSSL"). Two of the most controversal 
questions were:

 - Do we really need a new RNG API? Should the RAND_DRBG API be made public 
or kept private? (Currently, it's exported from libcrypto but only used 
internally by libssl.)
 - How much control should the user (programmer) be given over the 
reseeding process and/or should he be allowed to add his own additional 
randomness?

Many developers seem to be realizing the interesting possibilities of the 
DRBG API and are asking for public access to this new and promising API. One of 
the driving forces behind it is the question about how to do seeding and 
reseeding right. Among others, Uri Blumenthal asked for making the DRBG API 
public.

Currently, the OpenSSL core members seem to be reluctant to make the API 
public, at least at this early stage. I understand Rich Salz's viewpoint that 
this requires a thorough discussion, because a public interface can't be easil

Re: [openssl-dev] Plea for a new public OpenSSL RNG API

[Dr. Matthias St. Pierre]

> -Ursprüngliche Nachricht-
> Von: openssl-dev [mailto:openssl-dev-boun...@openssl.org] Im Auftrag von 
> Blumenthal, Uri - 0553 - MITLL
> Gesendet: Mittwoch, 30. August 2017 17:23
> An: openssl-dev@openssl.org
> Betreff: Re: [openssl-dev] Plea for a new public OpenSSL RNG API
>
> ...
> >  The only tricky part was to deal with temporary failures of the entropy 
> > source.
> 
> Did you experience that often? How did you deal with it?
> 

1) If the SmartCard is missing when our VPN service starts at boot time, the 
initial instantiation of the DRBG fails. This error is ignored, the DRBG 
remains uninstantiated.
2) If the SmartCard is pulled after the VPN connection is established and a 
rekey occurs, the DRBG fails, because it pulls entropy on every generate 
request and the entropy source fails. The NIST standard demands that this is a 
fatal error and the DRBG has to be uninstantiated. So we uninstantiate and our 
custom drbg_bytes() returns an error, which leads to a VPN disconnect.

In order to account for 1) and 2), in our custom drbg_bytes() implementation

3) we do a just-in-time (re-)instantiation of the DRBG before calling 
RAND_DRBG_generate().


Note that everything said above applies to the legacy setup of libcrypto 1.0.2x 
with FIPS DRBG.  In the future, we would use RAND_DRBG_generate() directly.


Regards, Matthias
 


smime.p7s
Description: S/MIME cryptographic signature
-- 
openssl-dev mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev

Re: [openssl-dev] Plea for a new public OpenSSL RNG API

[Blumenthal, Uri - 0553 - MITLL]

>> I would do exactly the opposite. “Normal” entropy is fetched from the 
default sources (/dev/urandom). But
>> when a sensitive (aka long-term) keys are generated, a (portable :) 
hardware RNG is plugged in and used with
>> RAND_add() equivalent. Reason – in my setup reliable trusted hardware 
RNG is not always on. If it were, I’d
>> use it as the main entropy source and be done with it.
>
>In general, I would agree with you. In our case, it was a requirement 
of the
>   government to trust only the SmartCard RNG. Since we use it for VPN
>   connections with SmartCard authentication this was no problem, because
>   the SmartCard must be present in order to initiate the IKEv2 exchange. 

Ah, that makes a lot of difference. 

>  The only tricky part was to deal with temporary failures of the entropy 
> source.

Did you experience that often? How did you deal with it?



smime.p7s
Description: S/MIME cryptographic signature
-- 
openssl-dev mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev

Re: [openssl-dev] Plea for a new public OpenSSL RNG API

[Dr. Matthias St. Pierre]

> >  We have a similar situation, on a small hardware device with little 
> own entropy
> > but with a smartcard reader.
> 
> Yes, but in most cases you cannot count on the smartcard (or smartcard-like 
> device) being in the reader.
> Which is why in my opinion this is an ideal case for “additional input” 
> source, but rather poor as the
> “main” entropy source.
> 
> ...
> 
>>  We implemented a get_entropy() call which fetches the entropy via 
> PKCS#11, and
>>  modified the rand_method such that RAND_DRBG_generate() is always 
> called with
>> prediction_resistance=1. So every generate() triggers a reseed(), the 
> entropy is fetched
>> from the smartcard and it is immediately postprocessed by the AES-CTR 
> DRBG.
>> The /dev/urandom device was only used as additional input.
>> So we didn't feel the need for an extra API call.
> 
> I would do exactly the opposite. “Normal” entropy is fetched from the default 
> sources (/dev/urandom). But
> when a sensitive (aka long-term) keys are generated, a (portable :) hardware 
> RNG is plugged in and used with
> RAND_add() equivalent. Reason – in my setup reliable trusted hardware RNG is 
> not always on. If it were, I’d
> use it as the main entropy source and be done with it.


In general, I would agree with you. In our case, it was a requirement of the 
government to trust only the SmartCard RNG. Since we use it for VPN connections 
with SmartCard authentication this was no problem, because the SmartCard must 
be present in order to initiate the IKEv2 exchange. The only tricky part was to 
deal with temporary failures of the entropy source.

Matthias



smime.p7s
Description: S/MIME cryptographic signature
-- 
openssl-dev mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev

Re: [openssl-dev] Plea for a new public OpenSSL RNG API

[Blumenthal, Uri - 0553 - MITLL]

On 8/30/17, 00:59, "openssl-dev on behalf of Paul Dale" 
 wrote:

>My thoughts are that the new RNG API should be made public once it has
>been properly designed.  We've a chance to get this right, let's take the 
> time
>and make an effort to do so.  There is no rush.

Not quite. If there is an RNG involved in generating long-term keys now – users 
better be able to control/affect it now.

>   I also feel that having an engine interface to the new RNG API is 
> worthwhile.

+1

>   It allows hardware sources to be used via the same API.

I rather doubt this. For example, my smartcard (accessible via PKCS#11) is a 
hardware source, which I occasionally use. How do you see it used with the same 
API?

>I would like to see an entropy argument to the equivalent to RAND_add.
>   Anyone is welcome to always pass zero in but there should be the option to 
> not. 
>   Consider an hardware source with _provable_ output quality, why shouldn't 
> it be
>   allowed to meaningfully contribute?

What’s the purpose of passing the entropy argument? How is the callee (the RNG) 
going to use it? Why should the OpenSSL code in general trust the received 
value (how can OpenSSL tell that the received randomness is indeed from a 
hardware source with provable output quality)? Finally, what does it matter? 

And they all “meaningfully contribute”. The only question is whether this 
contribution should prevent RNG from acquiring more entropy from other sources. 
My opinion is resounding no.

>   I like the idea of two independent global RNGs.

+1  ;-)

>  This does increase seeding requirements however.

If you can seed one, you can seed two.



-Original Message-
From: Dr. Matthias St. Pierre [mailto:matthias.st.pie...@ncp-e.com] 
Sent: Tuesday, 29 August 2017 7:45 PM
To: openssl-dev@openssl.org
Subject: [openssl-dev] Plea for a new public OpenSSL RNG API

Hi everybody,

on the [openssl-dev] mailing list, there has been a long ongoing discussion 
about the new RAND_DRBG API and comparing it with the old RAND_METHOD API (see 
"[openssl-dev] Work on a new RNG for OpenSSL"). Two of the most controversal 
questions were:

 - Do we really need a new RNG API? Should the RAND_DRBG API be made public 
or kept private? (Currently, it's exported from libcrypto but only used 
internally by libssl.)
 - How much control should the user (programmer) be given over the 
reseeding process and/or should he be allowed to add his own additional 
randomness?

Many developers seem to be realizing the interesting possibilities of the 
DRBG API and are asking for public access to this new and promising API. One of 
the driving forces behind it is the question about how to do seeding and 
reseeding right. Among others, Uri Blumenthal asked for making the DRBG API 
public.

Currently, the OpenSSL core members seem to be reluctant to make the API 
public, at least at this early stage. I understand Rich Salz's viewpoint that 
this requires a thorough discussion, because a public interface can't be easily 
changed and wrong decisions in the early phase can become a heavy burdon.

Nevertheless, I agree with Uri Blumenthal that the DRBG API should be made 
public. So here comes my

==
Plea for a new public OpenSSL RNG API:
==

The new RAND_DRBG is the superior API. It shouldn't be kept private and 
hidden behind the ancient RAND_METHOD API.
The philosophy of the two APIs is not very well compatible, in 
particular when it comes to reseeding and adding
additional unpredictable input. Hiding the RAND_DRBG behind the 
RAND_METHOD API only causes problems.
Also, it will force people to patch their OpenSSL copy if they want to 
use the superior API.

The RAND_DRBG API should become the new public OpenSSL RNG API and the 
old RAND_METHOD API should be deprecated
in the long run. This transition does not need to be rushed, but it 
would be good if there would be early consent
on the road map. I am thinking of a smooth transition with a phase of 
coexistence and a compatibility layer
mapping the default RAND_METHOD to the default public RAND_DRBG 
instance. (This compatibility layer already exists,
it's the 'RAND_OpenSSL()' method.)



Historical Background
=

As Rich already mentioned in his blog post, the RAND_DRBG isn't new. It's 
been a part of OpenSSL for a long time, hidden in the FIPS 2.0 Object Module.

I have been working with the FIPS DRBG for quite a while now, using a 
FIPS-capable OpenSSL 1.0.2x crypto library. The reason why our company switched 
to the FIPS DRBG is that one of our products runs on a small hardware device 
which does not have a reliable 

Re: [openssl-dev] Plea for a new public OpenSSL RNG API

[Dr. Matthias St. Pierre]

> -Ursprüngliche Nachricht-
> Von: openssl-dev [mailto:openssl-dev-boun...@openssl.org] Im Auftrag von Matt 
> Caswell
> Gesendet: Dienstag, 29. August 2017 16:36
> An: openssl-dev@openssl.org
> Betreff: Re: [openssl-dev] Plea for a new public OpenSSL RNG API
> 
> 
> 
> On 29/08/17 15:02, Salz, Rich via openssl-dev wrote:
> >
> > dev asap. If there are problems with it we can always revert it before
> > it makes it into a release.
> >
> > Someone on the OMC called that “flip-flopping” and seemed to be against 
> > this kind of thing.
> >
> > If we are going to have an additional API, then we should at least see a 
> > draft of the header file first.
> 
> Matthias, would you be willing to create such a draft as a github PR
> (marked as WIP for now)?
> 
> Matt

Oh thanks, that's a great honor to be asked to do this. I will give it a 
thought. Currently, the RAND_add()/RAND_bytes() problem needs to fixed first, 
so it will take a little while.

Matthias




smime.p7s
Description: S/MIME cryptographic signature
-- 
openssl-dev mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev

Re: [openssl-dev] Plea for a new public OpenSSL RNG API

[Dr. Matthias St. Pierre]

> I realize that reseed() not only mixes my “additional input” but also 
> replaces the entire state. NIST does
> not specify interface to “just” mix the “additional input” into the state 
> without replacing the whole state
> with some fresh entropy by calling Get_entropy_input(). Maybe we can provide 
> such a function call (that’s
> what I think RAND_add() is supposed to do), but I’m not certain here…

The reseed() operation pulls fresh entropy, but the old state is not discarded. 
Instead, the fresh entropy is mixed in, just like the additional input (and the 
reseed_counter is reset). Have a look at ctr_update() and ctr_df() and you will 
see that the 'entropy' and the 'adin' input are treated the same way to change 
the internal state. The derivation function treats 'entropy' and 'adin' 
equally, it's just a difference in interpretation (trusted vs. untrusted 
randomness).   

The NIST document has also a lot of nice graphics to explain the inner 
workings. This helped me a lot to understand the details.


Matthias




smime.p7s
Description: S/MIME cryptographic signature
-- 
openssl-dev mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev

Re: [openssl-dev] Plea for a new public OpenSSL RNG API

[Blumenthal, Uri - 0553 - MITLL]

>> I *don’t want* OpenSSL to make *any* estimation of the amount of provided 
>> entropy. All I want it to do is to mix these bits into the RNG state. It’s 
>> *my* business how much entropy I’m providing – but I don’t want OpenSSL to 
>> make any decision regarding pull from other entropy sources based on my 
>> input.
>> 
>> Does it sound reasonable? (He, it does to me ;)
> 
> But that is not the API that RAND_add() provides.

Yes, I realize that, sort of. But nonetheless, that’s the interface/behavior 
that I think an RNG should offer.

> It's a push not
> a pull API. With the DRBG you can do this, assuming using it's an
> extraction / derivative function.

Not sure I understand. But “…you can do this…” sounds encouraging. ;-)

> One of the suggestions I did before is to have RAND_poll_ex() take
> a parameter about how much randomness is needed, but I think it's
> also a wrong API and I'm thinking about removing it.

There are two aspects:

The RNG itself keeping track of how much entropy it has in its pool/state, and 
thus when to pull its (standard) entropy sources for more (and specifically for 
how much more). I don’t want to interfere with this part.
The user-visible/accessible interface to RNG that allows:
Forcing the “reseed”, making the RNG to to its sources for “fresh” entropy 
regardless of what’s up with its internal state;
Telling the RNG to immediately mix the given bits into its state, without 
adjusting its count of available/present entropy.



>> If I had my way, you’d assume that every bit contains 0 bits of entropy, but 
>> mix it in regardless because that’s what the user is asking you to do.
> 
> Which is why I suggested we use this for the additional data.

Yes, precisely. That’s exactly what I plan to use this interface for - mixing 
in additional data. NIST 800-90A Rev1 has “additional input” only as part of 
Reseed() or Generate(). I can live with that (i.e., with the fact that right 
before my randomness is mixed in, the entire state s replaced). Though ideally 
we’d have “RAND_add_ex()” or such that does not require replacing the entire 
state with fresh data from entropy source (because if that capability is 
needed, it is easy to emulate by issuing two calls: “Reseed(); Add_ex();"


> But
> I think that as long as we have both APIs we might actually need
> it for the entropy input. If there is no other way to add
> randomness, RAND_add() is our current documented way to add it,
> and it will need to keep working.

It’s perfectly OK for me if RAND_add() keeps working. What I’d like to change 
about it is exactly when the mixing occurs. I’d like the state updated 
immediately upon RAND_add(), not when the RNG in question decides that it needs 
to replenish its entropy supply. I think this change would only improve 
security of applications that use it.

TNX!-- 
openssl-dev mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev

Re: [openssl-dev] Plea for a new public OpenSSL RNG API

[Salz, Rich via openssl-dev]

➢ > Sure I can.  Because the DRBG seeds from the system anyway

➢ You can't assume that will work for all users.

And for places where the systesm doen’t have enough randomness, there is 
nothing we can do.

-- 
openssl-dev mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev

Re: [openssl-dev] Plea for a new public OpenSSL RNG API

[Kurt Roeckx]

On Tue, Aug 29, 2017 at 08:38:09PM +, Blumenthal, Uri - 0553 - MITLL wrote:
> > If, based on its value, OpenSSL may decide that it now got “enough” entropy 
> > and doesn’t need to
> > pull more from other sources before serving randomness to requestors – then 
> > it is harmful.
> > “Over-confidence” in this value by the caller can negatively impact the 
> > quality of the produced
> > random numbers.
> 
> As long as you have sources that don't provide 1 bit of randomness
> per bit that you provide you need to have an estimate of how much
> randomness it really contains. And you should probably seriously
> underestimate it so that you're sure that you collect enough.
> 
> So let me underestimate it to 0. ;-)  
>   
> The problem with ignoring an existing parameter is that people
> could be calling it with for instance the value of 0, knowing it
> contains as good as none entropy. Or they could feed the unwithened
> output of an TRNG in that with an estimate of randomness it provides.
> And OpenSSL used to do the right thing with that.
> 
> I *don’t want* OpenSSL to make *any* estimation of the amount of provided 
> entropy. All I want it to do is to mix these bits into the RNG state. It’s 
> *my* business how much entropy I’m providing – but I don’t want OpenSSL to 
> make any decision regarding pull from other entropy sources based on my input.
> 
> Does it sound reasonable? (He, it does to me ;)

But that is not the API that RAND_add() provides. It's a push not
a pull API. With the DRBG you can do this, assuming using it's an
extraction / derivative function.

One of the suggestions I did before is to have RAND_poll_ex() take
a parameter about how much randomness is needed, but I think it's
also a wrong API and I'm thinking about removing it.

> But now we just ignore it and assume every bit with get contains 1
> bit of randomness and we're sundenly seriously overestimating the
> amount of randomness we're getting.
> 
> If I had my way, you’d assume that every bit contains 0 bits of entropy, but 
> mix it in regardless because that’s what the user is asking you to do.

Which is why I suggested we use this for the additional data. But
I think that as long as we have both APIs we might actually need
it for the entropy input. If there is no other way to add
randomness, RAND_add() is our current documented way to add it,
and it will need to keep working.


Kurt

-- 
openssl-dev mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev

Re: [openssl-dev] Plea for a new public OpenSSL RNG API

[Salz, Rich via openssl-dev]

➢ But now we just ignore it and assume every bit with get contains 1
➢ bit of randomness and we're sundenly seriously overestimating the
➢ amount of randomness we're getting. This is a documented public API,
➢ you can't just go and ignore this parameter.

Sure I can.  Because the DRBG seeds from the system anyway

-- 
openssl-dev mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev

Re: [openssl-dev] Plea for a new public OpenSSL RNG API

[Blumenthal, Uri - 0553 - MITLL]

> If, based on its value, OpenSSL may decide that it now got “enough” entropy 
> and doesn’t need to
> pull more from other sources before serving randomness to requestors – then 
> it is harmful.
> “Over-confidence” in this value by the caller can negatively impact the 
> quality of the produced
> random numbers.

As long as you have sources that don't provide 1 bit of randomness
per bit that you provide you need to have an estimate of how much
randomness it really contains. And you should probably seriously
underestimate it so that you're sure that you collect enough.

So let me underestimate it to 0. ;-)  
  
The problem with ignoring an existing parameter is that people
could be calling it with for instance the value of 0, knowing it
contains as good as none entropy. Or they could feed the unwithened
output of an TRNG in that with an estimate of randomness it provides.
And OpenSSL used to do the right thing with that.

I *don’t want* OpenSSL to make *any* estimation of the amount of provided 
entropy. All I want it to do is to mix these bits into the RNG state. It’s *my* 
business how much entropy I’m providing – but I don’t want OpenSSL to make any 
decision regarding pull from other entropy sources based on my input.

Does it sound reasonable? (He, it does to me ;)

But now we just ignore it and assume every bit with get contains 1
bit of randomness and we're sundenly seriously overestimating the
amount of randomness we're getting.

If I had my way, you’d assume that every bit contains 0 bits of entropy, but 
mix it in regardless because that’s what the user is asking you to do.


This is a documented public API, you can't just go and ignore this 
parameter.

:-)



smime.p7s
Description: S/MIME cryptographic signature
-- 
openssl-dev mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev

Re: [openssl-dev] Plea for a new public OpenSSL RNG API

[Kurt Roeckx]

On Tue, Aug 29, 2017 at 06:50:37PM +, Blumenthal, Uri - 0553 - MITLL wrote:
> On 8/29/17, 12:45, "openssl-dev on behalf of Salz, Rich via openssl-dev" 
>  wrote:
> 
> ➢ An other problem with the current implemenation is that the
> ➢ randomness parameter that's now given to RAND_add() is just
> ➢ ignored, it assumes it's the same as the length.
> 
> For what it’s worth, this was done deliberately, make RAND_add and 
> RAND_seed equivalent.
> 
> I am skeptical of the ability to get that estimate correct.
> 
> Someone on GH there is a conversation thread about turning that into a 
> percentage, which seems like the best thing to do for any new API.
> 
> 
>  What’s the point of having this potentially harmful parameter? If it weren’t 
> ignored – how would OpenSSL use it?
> 
> If, based on its value, OpenSSL may decide that it now got “enough” entropy 
> and doesn’t need to pull more from other sources before serving randomness to 
> requestors – then it is harmful. “Over-confidence” in this value by the 
> caller can negatively impact the quality of the produced random numbers.

As long as you have sources that don't provide 1 bit of randomness
per bit that you provide you need to have an estimate of how much
randomness it really contains. And you should probably seriously
underestimate it so that you're sure that you collect enough.

The problem with ignoring an existing parameter is that people
could be calling it with for instance the value of 0, knowing it
contains as good as none entropy. Or they could feed the unwithened
output of an TRNG in that with an estimate of randomness it provides.
And OpenSSL used to do the right thing with that.

But now we just ignore it and assume every bit with get contains 1
bit of randomness and we're sundenly seriously overestimating the
amount of randomness we're getting. This is a documented public API,
you can't just go and ignore this parameter.


Kurt

-- 
openssl-dev mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev

Re: [openssl-dev] Plea for a new public OpenSSL RNG API

[Blumenthal, Uri - 0553 - MITLL]

On 8/29/17, 15:22, "openssl-dev on behalf of Salz, Rich via openssl-dev" 
 wrote:
➢ I’d like to suggest that any approach other than “immediately mix the 
received randomness into the RNG state” is bad. If a user does RAND_add() now, 
as opposed to 100 source code lines before, there may be a reason for that.

>I think the only way to do that in the DRBG model is to treat it as 
> “additional input” and
>do a generate call.  I think that would achieve the same effect, but it 
> wouldn’t delay any
>possible seeding of randomness that the DRBG itself needs at some point.
   
That is OK if the application manually invokes generate(). If it’s the RNG 
responsible for feeding key generators (who would call generate() transparently 
to user), then the only way I see is DRBG reseed(). Even though it implies a 
complete state replacement.

AFAIK, there are three cases when randomness source(s) is pulled:

a. At the instantiation time (page 29 of NIST SP 800-90A Rev1 draft).
b. Upon reseed call (pages 30-31).
c. During generation call(pages 32-36).

“Additional input” (i.e., what I’m after :) is present in two of the above: 
reseeding, and generation (pages 19 and 23).

The draft is very clear that this “additional input” shall not be counted 
towards the “official” entropy (aka calculations when to pull the “official” 
entropy sources). Which suits my purposes just fine. ;-)

I realize that reseed() not only mixes my “additional input” but also replaces 
the entire state. NIST does not specify interface to “just” mix the “additional 
input” into the state without replacing the whole state with some fresh entropy 
by calling Get_entropy_input(). Maybe we can provide such a function call 
(that’s what I think RAND_add() is supposed to do), but I’m not certain here… 

Wonder what others think? Do you believe that if you want to mix some more 
randomness into the RNG state to improve upon less-than-perfect entropy 
sources, you must also replace the entire state with the fresh pull from those 
sources? Or is it OK to just “spice” the current state with “additional input”?

Note that regardless of the answer to the above, IMHO reseed() should be able 
to take “additional input”. (NIST considers it optional, but sensitive/critical 
applications don’t. ;)

➢One “API” is how to get a reference/pointer/instance/whatever to the 
DRBG object responsible for the operation I’m now concerned with,
➢  and that I want to influence/improve. This would probably differ between 
per-SSL DRBG and per-thread DRBG, etc. I haven’t even
➢  thought about this part of the API (and I’m sure others on the team have 
more experience and understanding of the OpenSSL code
➢  to do it better than I would anyway).

 > Yes, it is separate.  But I want to make sure that if we are doing a 
 > comprehensive RAND overhaul that this is included in the requirements.

Yes, it should be included. But don’t let it stop us from defining/reviewing 
the “clearer” part of the API.  ;-)


smime.p7s
Description: S/MIME cryptographic signature
-- 
openssl-dev mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev

Re: [openssl-dev] Plea for a new public OpenSSL RNG API

[Richard Levitte]

In message <168cef3c-d655-47bf-9874-048308154...@ll.mit.edu> on Tue, 29 Aug 
2017 19:04:48 +, "Blumenthal, Uri - 0553 - MITLL"  said:

uri> uri> Could you please be more specific wrt. DRBG organization that in 
your opinion could impact the UI? 
uri> 
uri> Are you talking about the UI API or something else?
uri> 
uri> No-no-no, just the UI API. 
uri> 
uri> I used the term “UI” to emphasize that this is the “public” part of the 
API, exposed to the user, expected to be called by the user, and (probably) 
designed to be stable for some (hopefully considerable) time.

Ah, ok.  In an OpenSSL context, this gets a bit confusing since there
is an API called UI (  ).

-- 
Richard Levitte levi...@openssl.org
OpenSSL Project http://www.openssl.org/~levitte/
-- 
openssl-dev mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev

Re: [openssl-dev] Plea for a new public OpenSSL RNG API

[Salz, Rich via openssl-dev]

➢ I’d like to suggest that any approach other than “immediately mix the 
received randomness into the RNG state” is bad. If a user does RAND_add() now, 
as opposed to 100 source code lines before, there may be a reason for that.

I think the only way to do that in the DRBG model is to treat it as “additional 
input” and do a generate call.  I think that would achieve the same effect, but 
it wouldn’t delay any possible seeding of randomness that the DRBG itself needs 
at some point.


➢ One “API” is how to get a reference/pointer/instance/whatever to the DRBG 
object responsible for the operation I’m now concerned with, and that I want to 
influence/improve. This would probably differ between per-SSL DRBG and 
per-thread DRBG, etc. I haven’t even thought about this part of the API (and 
I’m sure others on the team have more experience and understanding of the 
OpenSSL code to do it better than I would anyway).

  Yes, it is separate.  But I want to make sure that if we are doing a 
comprehensive RAND overhaul that this is included in the requirements.

-- 
openssl-dev mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev

Re: [openssl-dev] Plea for a new public OpenSSL RNG API

[Blumenthal, Uri - 0553 - MITLL]

On 8/29/17, 11:33, "openssl-dev on behalf of Salz, Rich via openssl-dev" 
 wrote:

Could you please be more specific wrt. DRBG organization that in your 
opinion could impact the UI? 

 >   From your use-case:  you want to add entropy into a specific DRBG. 

Yes.

  >  You want to push it, as opposed to the DRBG “pull when needed” model.  

I’d like to suggest that any approach other than “immediately mix the received 
randomness into the RNG state” is bad. If a user does RAND_add() now, as 
opposed to 100 source code lines before, there may be a reason for that.

Would you agree? Besides possible but probably negligible performance 
difference – why would you ever want to delay mixing the received “additional 
input” in?

  >  That’s an additional API.  

I wouldn’t call it “additional”. It may be a change from the current behavior – 
but I think everybody would welcome such a change. IMHO it can only help, never 
hurt.

   >  Also from your use-case: you want to specify which DRBG instance gets 
that entropy. 

Yeah, but that probably isn’t a part of the API that DRBG “object” exposes. 

One “API” is how to get a reference/pointer/instance/whatever to the DRBG 
object responsible for the operation I’m now concerned with, and that I want to 
influence/improve. This would probably differ between per-SSL DRBG and 
per-thread DRBG, etc. I haven’t even thought about this part of the API (and 
I’m sure others on the team have more experience and understanding of the 
OpenSSL code to do it better than I would anyway).

Another “API” is how to tell this specific DRBG instance what exactly I want 
from it now. E.g., mix more randomness that I provide into its state, give me 
some random bits, whatever. This part probably can stay close to what it 
currently is. I think 90A would be satisfied with 3-4 interface calls here…

   >   If we move to a pair per thread, as opposed to one per SSL and two in 
the global space, how do we make sure that API still works and does the right 
thing.

Yeah. That’s the “other” part of the API. I think the two API “groups” are 
pretty (completely?) orthogonal – independent from each other.

   >Does that makes sense, and does it answer your question?

 Yeah… What do you think of my reasoning above?



smime.p7s
Description: S/MIME cryptographic signature
-- 
openssl-dev mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev

Re: [openssl-dev] Plea for a new public OpenSSL RNG API

[Blumenthal, Uri - 0553 - MITLL]

IMHO this interface is a way for the user to improve the quality of the 
randomness it would get from the given RNG, *not* to replace (or diminish) its 
other sources. My proposal is to abolish this parameter, especially since now 
it is simply ignored (and IMHO – for a good reason).

That's a fine proposal ... it just can't be implemented until a major release 
boundary, when our ABI stability policy permits such breaking changes.


And that is fine. The sooner the better, but ABI stability makes sense too. 

 

My main point is:  RAND_add() and whatever similar in purpose interface calls 
we may define in the future should exhibit the following behavior:

 
Mix the provided randomness into the RNG state *immediately*, and
Keep pulling other sources and mixing them into the state – don’t subtract from 
the “needed entropy” count the amount you presumably got from the user.


Frankly, the need to provide double entropy argument doesn’t bother me all that 
much – especially if the value 0 is accepted there. ;-)



smime.p7s
Description: S/MIME cryptographic signature
-- 
openssl-dev mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev

Re: [openssl-dev] Plea for a new public OpenSSL RNG API

[Blumenthal, Uri - 0553 - MITLL]

uri> Could you please be more specific wrt. DRBG organization that in your 
opinion could impact the UI? 

Are you talking about the UI API or something else?

No-no-no, just the UI API. 

I used the term “UI” to emphasize that this is the “public” part of the API, 
exposed to the user, expected to be called by the user, and (probably) designed 
to be stable for some (hopefully considerable) time.



smime.p7s
Description: S/MIME cryptographic signature
-- 
openssl-dev mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev

Re: [openssl-dev] Plea for a new public OpenSSL RNG API

[Salz, Rich via openssl-dev]

➢ IMHO this interface is a way for the user to improve the quality of the 
randomness it would get from the given RNG, *not* to replace (or diminish) its 
other sources. My proposal is to abolish this parameter, especially since now 
it is simply ignored (and IMHO – for a good reason).

We cannot do this in a minor release; we have to wait until 1.2




-- 
openssl-dev mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev

Re: [openssl-dev] Plea for a new public OpenSSL RNG API

[Benjamin Kaduk via openssl-dev]

On 08/29/2017 01:50 PM, Blumenthal, Uri - 0553 - MITLL wrote:
> IMHO this interface is a way for the user to improve the quality of the 
> randomness it would get from the given RNG, *not* to replace (or diminish) 
> its other sources. My proposal is to abolish this parameter, especially since 
> now it is simply ignored (and IMHO – for a good reason).

That's a fine proposal ... it just can't be implemented until a major
release boundary, when our ABI stability policy permits such breaking
changes.

-Ben
-- 
openssl-dev mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev

Re: [openssl-dev] Plea for a new public OpenSSL RNG API

[Blumenthal, Uri - 0553 - MITLL]

On 8/29/17, 12:45, "openssl-dev on behalf of Salz, Rich via openssl-dev" 
 wrote:

➢ An other problem with the current implemenation is that the
➢ randomness parameter that's now given to RAND_add() is just
➢ ignored, it assumes it's the same as the length.

For what it’s worth, this was done deliberately, make RAND_add and 
RAND_seed equivalent.

I am skeptical of the ability to get that estimate correct.

Someone on GH there is a conversation thread about turning that into a 
percentage, which seems like the best thing to do for any new API.


 What’s the point of having this potentially harmful parameter? If it weren’t 
ignored – how would OpenSSL use it?

If, based on its value, OpenSSL may decide that it now got “enough” entropy and 
doesn’t need to pull more from other sources before serving randomness to 
requestors – then it is harmful. “Over-confidence” in this value by the caller 
can negatively impact the quality of the produced random numbers.

If this value is not used to guide OpenSSL when to stop pulling entropy sources 
and start serving randomness – then it causes no harm, but what’s its purpose?

IMHO this interface is a way for the user to improve the quality of the 
randomness it would get from the given RNG, *not* to replace (or diminish) its 
other sources. My proposal is to abolish this parameter, especially since now 
it is simply ignored (and IMHO – for a good reason).




smime.p7s
Description: S/MIME cryptographic signature
-- 
openssl-dev mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev

Re: [openssl-dev] Plea for a new public OpenSSL RNG API

[Salz, Rich via openssl-dev]

➢ An other problem with the current implemenation is that the
➢ randomness parameter that's now given to RAND_add() is just
➢ ignored, it assumes it's the same as the length.

For what it’s worth, this was done deliberately, make RAND_add and RAND_seed 
equivalent.

I am skeptical of the ability to get that estimate correct.

Someone on GH there is a conversation thread about turning that into a 
percentage, which seems like the best thing to do for any new API.


-- 
openssl-dev mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev

Re: [openssl-dev] Plea for a new public OpenSSL RNG API

[Kurt Roeckx]

On Tue, Aug 29, 2017 at 01:05:21PM +, Dr. Matthias St. Pierre wrote:
> 
> Currently it's only possible to customize the callbacks but not the 
> deterministic algorithm. IMHO this is sufficient for the needs of a standard 
> OpenSSL user who only wants control over the entropy source. A true new 
> algorithm (like e.g. CHACHA2_DRBG) should be implemented by experts and added 
> mainstream. So I don't see any advantage of having an engine over using the 
> 'vtable' approach from the FIPS DRBG, which has been removed.

I've been looking at implementing a chacha20 based DRBG, which
isn't that hard. But there are various choices you can make, and
every chacha20 RNG that I've seen seems to take a random set of
those choices. I really wish there was some standardized version.


Kurt

-- 
openssl-dev mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev

Re: [openssl-dev] Plea for a new public OpenSSL RNG API

[Kurt Roeckx]

On Tue, Aug 29, 2017 at 11:31:03AM +, Dr. Matthias St. Pierre wrote:
> > -Ursprüngliche Nachricht-
> > Von: openssl-dev [mailto:openssl-dev-boun...@openssl.org] Im Auftrag von 
> > Matt Caswell
> > Gesendet: Dienstag, 29. August 2017 12:17
> > An: openssl-dev@openssl.org
> > Betreff: Re: [openssl-dev] Plea for a new public OpenSSL RNG API
> > 
> > 
> > On 29/08/17 10:45, Dr. Matthias St. Pierre wrote
> > > ...
> > > The 'RAND_add()/RAND_bytes()' pattern is broken
> > > ===
> > >
> > > In OpenSSL, the classical way for the RNG consumer to add his own
> > > randomness is to call 'RAND_add()' before calling 'RAND_bytes()'. If
> > > the new 'RAND_OpenSSL()' method (the "compatibility layer" hiding the
> > > public RAND_DRBG instance)  is the default, then this does not work
> > > as expected anymore:
> > > ...
> > 
> > Is there a potential security vulnerability here? Applications using the
> > "old" APIs expect RAND_add() to behave in a particular way. If we have
> > silently changed this behaviour in 1.1.1 are they exposed?
> 
> Don't worry, this issue is new, the global 'rand_bytes' buffer has only been 
> introduced by the DRBG port to master in August. I don't think it's a big 
> deal to fix it. The reason I mentioned it here was to emphasize, that it is 
> really hard to get the different philosophies (push vs. pull) of the two APIs 
> working together correctly. The code was reviewed by several people and 
> nobody noticed it. By the way: the approach using the fixed size global 
> 'rand_bytes' buffer has another issue, which I will try to write down on 
> GitHub within the next days.

I've actually noticed how this works and I have already partially
rewritten it, but I'm still not very happy about it. I think by
RAND_add() should not be called internally. But the question then
is what to do when an application calls RAND_add(), we should be
doing something with the buffer that's given. I think the best way
to deal with it is with the DRBG API is used, RAND_add() is used for
additional data.

We now have 2 global DRBGs, and I think I want to have 1 of them
chain to the other. RAND_add() could then be used for the master.

An other problem with the current implemenation is that the
randomness parameter that's now given to RAND_add() is just
ignored, it assumes it's the same as the length.


Kurt

-- 
openssl-dev mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev

Re: [openssl-dev] Plea for a new public OpenSSL RNG API

[Kurt Roeckx]

I actually plan to work on most of not all the things you mention.
This will probably result in the API changing before it's made
public. I'm probably slow, so please be patient.


Kurt

-- 
openssl-dev mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev

Re: [openssl-dev] Plea for a new public OpenSSL RNG API

[Salz, Rich via openssl-dev]

Could you please be more specific wrt. DRBG organization that in your 
opinion could impact the UI? 

From your use-case:  you want to add entropy into a specific DRBG.  You want to 
push it, as opposed to the DRBG “pull when needed” model.  That’s an additional 
API.  Also from your use-case: you want to specify which DRBG instance gets 
that entropy.  If we move to a pair per thread, as opposed to one per SSL and 
two in the global space, how do we make sure that API still works and does the 
right thing.

Does that makes sense, and does it answer your question?


-- 
openssl-dev mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev

Re: [openssl-dev] Plea for a new public OpenSSL RNG API

[Blumenthal, Uri - 0553 - MITLL]

Could you please be more specific wrt. DRBG organization that in your opinion 
could impact the UI? 

NIST 90Ar1 seems specific enough on what functionality DRBG can provide to 
users, and it doesn't seem like a very elaborate or rich interface. Why would 
it change, regardless of what you may need to do with it internally?

Regards,
Uri

Sent from my iPhone

> On Aug 29, 2017, at 10:03, Salz, Rich via openssl-dev 
>  wrote:
> 
> 
>dev asap. If there are problems with it we can always revert it before
>it makes it into a release.
> 
> Someone on the OMC called that “flip-flopping” and seemed to be against this 
> kind of thing.
> 
> If we are going to have an additional API, then we should at least see a 
> draft of the header file first.
> 
> Keep in mind that the current DRBG organization might change, and we don’t 
> want to lose that freedom.
> 
> 
> 
> -- 
> openssl-dev mailing list
> To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev


smime.p7s
Description: S/MIME cryptographic signature
-- 
openssl-dev mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev

Re: [openssl-dev] Plea for a new public OpenSSL RNG API

[Matt Caswell]

On 29/08/17 15:02, Salz, Rich via openssl-dev wrote:
> 
> dev asap. If there are problems with it we can always revert it before
> it makes it into a release.
>  
> Someone on the OMC called that “flip-flopping” and seemed to be against this 
> kind of thing.
> 
> If we are going to have an additional API, then we should at least see a 
> draft of the header file first.

Matthias, would you be willing to create such a draft as a github PR
(marked as WIP for now)?

Matt
-- 
openssl-dev mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev

Re: [openssl-dev] Plea for a new public OpenSSL RNG API

[Salz, Rich via openssl-dev]

dev asap. If there are problems with it we can always revert it before
it makes it into a release.
 
Someone on the OMC called that “flip-flopping” and seemed to be against this 
kind of thing.

If we are going to have an additional API, then we should at least see a draft 
of the header file first.

Keep in mind that the current DRBG organization might change, and we don’t want 
to lose that freedom.



-- 
openssl-dev mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev

Re: [openssl-dev] Plea for a new public OpenSSL RNG API

[Richard Levitte]

In message <3a54fcefd17d4735920426d21b3b8...@ex13.ncp.local> on Tue, 29 Aug 
2017 13:39:02 +, "Dr. Matthias St. Pierre"  
said:

Matthias.St.Pierre> Just a sudden inspiration: If the RAND_DRBG becomes a truly 
independent API it might be better to strip the RAND_ prefix and redesign the 
API such that one has
Matthias.St.Pierre> 
Matthias.St.Pierre> - a DRBG_CTX structure for the data members
Matthias.St.Pierre> - a DRBG_METHOD  structure for its methods
Matthias.St.Pierre> 
Matthias.St.Pierre> Would this look more OpenSSL-like to you?

Yes.  And per fairly recent recommendations to avoid cluttering the
name space, that would be OSSL_DRGB_CTX and OSSL_DRGB_METHOD, btw.

-- 
Richard Levitte levi...@openssl.org
OpenSSL Project http://www.openssl.org/~levitte/
-- 
openssl-dev mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev

Re: [openssl-dev] Plea for a new public OpenSSL RNG API

[Dr. Stephen Henson]

On Tue, Aug 29, 2017, Richard Levitte wrote:

> I'm late in the game, having only followed the development very
> superficially...
> 
> If I understand correctly, the RAND_DRBG API is really a completely
> separate API that has nothing to do with the RAND_METHOD API pers se,
> i.e. any association between the two is more or less "accidental"?
> 
> Frankly, I cannot see anything wrong with making that a public API.  I
> wonder, though, if this is going to be an implementation that's kinda
> sorta built-in only, or if other implementations of the basic stuff
> will be possible...  or in other words, will we have a "method"
> structure like we have with so many other APIs?  In saying this, I'm
> counting in the possibility that some implementations could come in
> the form of engines, is this something that's been thought about?  I
> do notive the DRGB_CTX structure, but that doesn't quite follow the
> usual "method" pattern we have...
> 

In terms of future FIPS 140-2 compliance having a way to redirect all PRNG
operations to a FIPS compliant version is essential. This would require some
kind of method. We can redirect via RAND_METHOD but that currently means that
any ENGINE supplied PRNG is severely restricted (one global PRNG only)
compared to the built in DRBG. We could resolve that via a DRBG_METHOD or some
extended RAND_METHOD.

That's for the future though: there wont be a new FIPS module for some time
that will need this functionality.

In terms of the API itself. We could make RAND_DRBG public and I can see
reasons both for and against that. The main objection I can see is that it
commits us to DRBG like PRNGs: that is if we (or an ENGINE) in future wants to
use a RNG that isn't a good fit for the DRBG API it's a problem. We're
partially hitting that now because the DRBG API is not a good fit for the
older RAND API.

One alternative is to expand RAND_METHOD so it treats a DRBG as a special case
of the expanded API and it is flexible enough to handle any future needs. That
would (for example) have extended RAND APIs which can be passed an "instance"
of a PRNG (something like RAND_CTX which is analagous to the DRBG instance
structure we currently have).

That is tricky because RAND_METHOD is alas not an opaque structure and can't
be made so before OpenSSL 1.2.0. Tricky but not impossible.

Steve.
--
Dr Stephen N. Henson. OpenSSL project core developer.
Commercial tech support now available see: http://www.openssl.org
-- 
openssl-dev mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev

Re: [openssl-dev] Plea for a new public OpenSSL RNG API

[Richard Levitte]

In message  on Tue, 29 Aug 
2017 13:27:20 +, "Dr. Matthias St. Pierre"  
said:

Matthias.St.Pierre> > Essentially, the argument for your last remark is 
in-structure vtable
Matthias.St.Pierre> > vs refered to vtable.  I tend to prefer the latter (and 
that's the
Matthias.St.Pierre> > usual OpenSSL pattern too, even though there are 
exceptions).
Matthias.St.Pierre> 
Matthias.St.Pierre> You are the experts and much more familiar with
Matthias.St.Pierre> the code then I am. My role was only to give the
Matthias.St.Pierre> starting shot, the rest is up to you.

Fair enough!  :-)

-- 
Richard Levitte levi...@openssl.org
OpenSSL Project http://www.openssl.org/~levitte/
-- 
openssl-dev mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev

Re: [openssl-dev] Plea for a new public OpenSSL RNG API

[Dr. Matthias St. Pierre]

> Essentially, the argument for your last remark is in-structure vtable
> vs refered to vtable.  I tend to prefer the latter (and that's the
> usual OpenSSL pattern too, even though there are exceptions).

You are the experts and much more familiar with the code then I am. My role was 
only to give the starting shot, the rest is up to you.

Regards,
Matthias




smime.p7s
Description: S/MIME cryptographic signature
-- 
openssl-dev mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev

Re: [openssl-dev] Plea for a new public OpenSSL RNG API

[Richard Levitte]

In message <0e062727611c44bb8039170bbec11...@ex13.ncp.local> on Tue, 29 Aug 
2017 13:05:21 +, "Dr. Matthias St. Pierre"  
said:

Matthias.St.Pierre> > Frankly, I cannot see anything wrong with making that a 
public API.  I
Matthias.St.Pierre> > wonder, though, if this is going to be an implementation 
that's kinda
Matthias.St.Pierre> > sorta built-in only, or if other implementations of the 
basic stuff
Matthias.St.Pierre> > will be possible...  or in other words, will we have a 
"method"
Matthias.St.Pierre> > structure like we have with so many other APIs?  In 
saying this, I'm
Matthias.St.Pierre> > counting in the possibility that some implementations 
could come in
Matthias.St.Pierre> > the form of engines, is this something that's been 
thought about?  I
Matthias.St.Pierre> > do notive the DRGB_CTX structure, but that doesn't quite 
follow the
Matthias.St.Pierre> > usual "method" pattern we have...
Matthias.St.Pierre> 
Matthias.St.Pierre> Currently it's only possible to customize the callbacks but 
not the deterministic algorithm. IMHO this is sufficient for the needs of a 
standard OpenSSL user who only wants control over the entropy source. A true 
new algorithm (like e.g. CHACHA2_DRBG) should be implemented by experts and 
added mainstream. So I don't see any advantage of having an engine over using 
the 'vtable' approach from the FIPS DRBG, which has been removed.

Essentially, the argument for your last remark is in-structure vtable
vs refered to vtable.  I tend to prefer the latter (and that's the
usual OpenSSL pattern too, even though there are exceptions).

-- 
Richard Levitte levi...@openssl.org
OpenSSL Project http://www.openssl.org/~levitte/
-- 
openssl-dev mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev

Re: [openssl-dev] Plea for a new public OpenSSL RNG API

[Richard Levitte]

I'm late in the game, having only followed the development very
superficially...

If I understand correctly, the RAND_DRBG API is really a completely
separate API that has nothing to do with the RAND_METHOD API pers se,
i.e. any association between the two is more or less "accidental"?

Frankly, I cannot see anything wrong with making that a public API.  I
wonder, though, if this is going to be an implementation that's kinda
sorta built-in only, or if other implementations of the basic stuff
will be possible...  or in other words, will we have a "method"
structure like we have with so many other APIs?  In saying this, I'm
counting in the possibility that some implementations could come in
the form of engines, is this something that's been thought about?  I
do notive the DRGB_CTX structure, but that doesn't quite follow the
usual "method" pattern we have...

Cheers,
Richard

In message  on Tue, 29 Aug 
2017 09:45:26 +, "Dr. Matthias St. Pierre"  
said:

Matthias.St.Pierre> Hi everybody,
Matthias.St.Pierre> 
Matthias.St.Pierre> on the [openssl-dev] mailing list, there has been a long 
ongoing discussion about the new RAND_DRBG API and comparing it with the old 
RAND_METHOD API (see "[openssl-dev] Work on a new RNG for OpenSSL"). Two of the 
most controversal questions were:
Matthias.St.Pierre> 
Matthias.St.Pierre>  - Do we really need a new RNG API? Should the RAND_DRBG 
API be made public or kept private? (Currently, it's exported from libcrypto 
but only used internally by libssl.)
Matthias.St.Pierre>  - How much control should the user (programmer) be given 
over the reseeding process and/or should he be allowed to add his own 
additional randomness?
Matthias.St.Pierre> 
Matthias.St.Pierre> Many developers seem to be realizing the interesting 
possibilities of the DRBG API and are asking for public access to this new and 
promising API. One of the driving forces behind it is the question about how to 
do seeding and reseeding right. Among others, Uri Blumenthal asked for making 
the DRBG API public.
Matthias.St.Pierre> 
Matthias.St.Pierre> Currently, the OpenSSL core members seem to be reluctant to 
make the API public, at least at this early stage. I understand Rich Salz's 
viewpoint that this requires a thorough discussion, because a public interface 
can't be easily changed and wrong decisions in the early phase can become a 
heavy burdon.
Matthias.St.Pierre> 
Matthias.St.Pierre> Nevertheless, I agree with Uri Blumenthal that the DRBG API 
should be made public. So here comes my
Matthias.St.Pierre> 
Matthias.St.Pierre> ==
Matthias.St.Pierre> Plea for a new public OpenSSL RNG API:
Matthias.St.Pierre> ==
Matthias.St.Pierre> 
Matthias.St.Pierre> The new RAND_DRBG is the superior API. It shouldn't be 
kept private and hidden behind the ancient RAND_METHOD API.
Matthias.St.Pierre> The philosophy of the two APIs is not very well 
compatible, in particular when it comes to reseeding and adding
Matthias.St.Pierre> additional unpredictable input. Hiding the RAND_DRBG 
behind the RAND_METHOD API only causes problems.
Matthias.St.Pierre> Also, it will force people to patch their OpenSSL copy 
if they want to use the superior API.
Matthias.St.Pierre> 
Matthias.St.Pierre> The RAND_DRBG API should become the new public OpenSSL 
RNG API and the old RAND_METHOD API should be deprecated
Matthias.St.Pierre> in the long run. This transition does not need to be 
rushed, but it would be good if there would be early consent
Matthias.St.Pierre> on the road map. I am thinking of a smooth transition 
with a phase of coexistence and a compatibility layer
Matthias.St.Pierre> mapping the default RAND_METHOD to the default public 
RAND_DRBG instance. (This compatibility layer already exists,
Matthias.St.Pierre> it's the 'RAND_OpenSSL()' method.)
Matthias.St.Pierre> 
Matthias.St.Pierre> 
Matthias.St.Pierre> 
Matthias.St.Pierre> Historical Background
Matthias.St.Pierre> =
Matthias.St.Pierre> 
Matthias.St.Pierre> As Rich already mentioned in his blog post, the RAND_DRBG 
isn't new. It's been a part of OpenSSL for a long time, hidden in the FIPS 2.0 
Object Module.
Matthias.St.Pierre> 
Matthias.St.Pierre> I have been working with the FIPS DRBG for quite a while 
now, using a FIPS-capable OpenSSL 1.0.2x crypto library. The reason why our 
company switched to the FIPS DRBG is that one of our products runs on a small 
hardware device which does not have a reliable entropy source, but the product 
has to meet high security standards, in particular w.r.t. its RNG. So we 
decided to use the SmartCard RNG as primary entropy source for a deterministic 
AES-CTR based RNG and use /dev/urandom as additional input. Reseeding should 
occur on every generate request. Using the FIPS DRBG, these requirements were 
easily met, because the API 

Re: [openssl-dev] Plea for a new public OpenSSL RNG API

[Dr. Matthias St. Pierre]

> -Ursprüngliche Nachricht-
> Von: openssl-dev [mailto:openssl-dev-boun...@openssl.org] Im Auftrag von Matt 
> Caswell
> Gesendet: Dienstag, 29. August 2017 12:17
> An: openssl-dev@openssl.org
> Betreff: Re: [openssl-dev] Plea for a new public OpenSSL RNG API
> 
> 
> On 29/08/17 10:45, Dr. Matthias St. Pierre wrote
> > ...
> > The 'RAND_add()/RAND_bytes()' pattern is broken
> > ===
> >
> > In OpenSSL, the classical way for the RNG consumer to add his own
> > randomness is to call 'RAND_add()' before calling 'RAND_bytes()'. If
> > the new 'RAND_OpenSSL()' method (the "compatibility layer" hiding the
> > public RAND_DRBG instance)  is the default, then this does not work
> > as expected anymore:
> > ...
> 
> Is there a potential security vulnerability here? Applications using the
> "old" APIs expect RAND_add() to behave in a particular way. If we have
> silently changed this behaviour in 1.1.1 are they exposed?

Don't worry, this issue is new, the global 'rand_bytes' buffer has only been 
introduced by the DRBG port to master in August. I don't think it's a big deal 
to fix it. The reason I mentioned it here was to emphasize, that it is really 
hard to get the different philosophies (push vs. pull) of the two APIs working 
together correctly. The code was reviewed by several people and nobody noticed 
it. By the way: the approach using the fixed size global 'rand_bytes' buffer 
has another issue, which I will try to write down on GitHub within the next 
days.

Matthias
 



smime.p7s
Description: S/MIME cryptographic signature
-- 
openssl-dev mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev

Re: [openssl-dev] Plea for a new public OpenSSL RNG API

[Matt Caswell]

On 29/08/17 10:45, Dr. Matthias St. Pierre wrote
> Currently, the OpenSSL core members seem to be reluctant to make the
> API public, at least at this early stage. I understand Rich Salz's
> viewpoint that this requires a thorough discussion, because a public
> interface can't be easily changed and wrong decisions in the early
> phase can become a heavy burdon.

FWIW, I am not against making the API public although I too understand
Rich's reluctance. However it seems inevitable to me that sooner or
later this API will have to be made accessible and I see little benefit
in delaying that and plenty of reasons for doing it now (it only
frustrates those that know the functionality is there but can't get at
it :-)). There is always a risk with every API added that you get it
wrong, and then you are stuck with it for ever more. The only real
answers to this are:

1) Make sure you "design" your API to start with and get it reviewed
2) Encourage people to try the API while it is still in dev i.e. so we
can still change it before the final release.

I'm not really convinced that keeping the API hidden now will mean that
it will be better in some future release. It just delays the point at
which we have the realisation of "if only we had designed it like *this*
it would have been s much better". Better is to make it public in
dev asap. If there are problems with it we can always revert it before
it makes it into a release.

> The 'RAND_add()/RAND_bytes()' pattern is broken 
> ===
> 
> In OpenSSL, the classical way for the RNG consumer to add his own
> randomness is to call 'RAND_add()' before calling 'RAND_bytes()'. If
> the new 'RAND_OpenSSL()' method (the "compatibility layer" hiding the
> public RAND_DRBG instance)  is the default, then this does not work
> as expected anymore:
> 
> The reason is that a call to 'RAND_add()' adds the provided
> randomness only to a global buffer ('rand_bytes'), from which it will
> be pulled during the next reseed. But no reseed is triggered. So the
> next RAND_bytes() call will be unaffected from the RAND_add(), which
> is not what the consumer expected. (The same holds for 'RAND_seed()',
> since 'drbg_seed()' only calls into 'drbg_add()')
> 
> Reseeding of DRBGs occurs only at the following occasions:
> 
> * immediately after a 'fork()' (new) * if the 'reseed_counter'
> exceeds the 'reseed_interval' * if 'RAND_DRBG_generate()' is called
> requesting 'prediction_resistance' * 'RAND_DRBG_reseed()' is called
> explicitely
> 
> *Note:* Currently it looks like the situation is even worse: if
> 'RAND_add()' is called multiple times before a reseed occurs, then
> the result of the previous call is overwritten.
> 
> Reseeding the 'DRBG' whenever the user calls 'RAND_add()' does not
> seem a good solution. It would be too expensive, in particular if
> system entropy is pulled for reseeding. Of course it is possible to
> fix this issue, but the DRBG provides for a much simpler solution: it
> lets the consumer contribute to the entropy of the internal state by
> providing additional input. If the user input contains entropy,
> that's fine, if it's "snake oil", no harm. The additional input is
> mixed into the internal state in just the same way as the entropy
> buffer using the 'ctr_df()' derivation function. One might think of
> the 'entropy' input as trusted randomness and 'adin' as untrusted
> randomness.
> 
> For this reason, I would like to see the 'RAND_add()/RAND_bytes()'
> pattern deprecated and the 'RAND_DRBG_generate() with additional
> input' pattern advertised instead.

Deprecation itself would probably have to wait for a 1.2.0 release. We
have a binary/source compatibility guarantee for 1.1.0 which would not
be met for builds using "--api" or "no-deprecated".

Is there a potential security vulnerability here? Applications using the
"old" APIs expect RAND_add() to behave in a particular way. If we have
silently changed this behaviour in 1.1.1 are they exposed?

Matt
-- 
openssl-dev mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev