Re: [openssl-project] OpenSSL 3.0 and FIPS Update

[Richard Levitte]

On Sat, 23 Feb 2019 21:47:00 +0100,
Dmitry Belyavsky wrote:
> 
> 
> Dear Richard, 
> 
> On Sat, Feb 23, 2019 at 8:47 AM Richard Levitte  wrote:
> 
> Since our RAND API is separate from the EVP API, I'm unsure how we
> plan on getting custom RAND_methods from providers.
>
> Please note that we can add RAND to the list of provider backed APIs,
> and given a foundation that we're currently building, it may even be
> quite easy.  However, no one has said explicitly that we would do so.
>
> The other option is, of course, to move the RAND API to EVP somehow,
> but that will probably be more challenging.
> 
> I do not think it is really necessary to move RAND to EVP.
> Current architecture suits our requirements, but if the possibility to 
> overwrite
> the RAND_METHOD is removed, it will cause problems for us.

So it turns out that some of my collegues were assuming that the RAND
API would be provider backed.  I simply hadn't caught on to that...

Cheers,
Richard

-- 
Richard Levitte levi...@openssl.org
OpenSSL Project http://www.openssl.org/~levitte/

Re: [openssl-project] OpenSSL 3.0 and FIPS Update

[Richard Levitte]

On Mon, 25 Feb 2019 00:40:51 +0100,
Michael Richardson wrote:
> I think that the #define/enum of NIDs should be made internal-only,
> available as optimization to internal code only.

Having asked around a bit on this, that was the original intention...
However, in an old era of having everything in public headers (or at
least everything that was of interest to the public *plus* everything
that libssl might want to use), they slipped out.

NID literally means "numeric identity" and really has no inherent
meaning other than quick access, like you say.

The public interface was meant to be getting stuff by name (string) or
possibly special functions such as EVP_aes_128_cbc()...

> Your question then becomes, "are engines internal users", and I'd like the
> answer to be no. I think that the openssl 3 changes suggest the same thing.

Yup.

> All other users can call OBJ_obj2nid() or OBJ_txt2nid() to get a NID,
> and we can figure out how to allocate things dynamically if this makes
> sense.  I don't know which APIs are currently NID-only.

There are some new APIs in master that add such functions:

EVP_MAC_CTX_new_id()
EVP_KDF_CTX_new_id()

I'm currently thinking that's a mistake.

Cheers,
Richard

-- 
Richard Levitte levi...@openssl.org
OpenSSL Project http://www.openssl.org/~levitte/

Re: [openssl-project] OpenSSL 3.0 and FIPS Update

[Dmitry Belyavsky]

Dear Dr Paul,

I think this change is somewhere in a gray zone.

On Mon, Feb 25, 2019 at 1:37 PM Dr Paul Dale  wrote:

> I don’t think that that new OIDs or NIDs are considering breaking.
> Changing existing ones definitely is, but that’s an entirely different
> proposition.
>
>
> Pauli
> --
> Dr Paul Dale | Cryptographer | Network Security & Encryption
> Phone +61 7 3031 7217
> Oracle Australia
>
>
>
> On 25 Feb 2019, at 5:02 pm, Dmitry Belyavsky  wrote:
>
>
>
> On Sun, Feb 24, 2019 at 11:31 PM Viktor Dukhovni <
> openssl-us...@dukhovni.org> wrote:
>
>> On Thu, Feb 21, 2019 at 04:20:53PM +, Matt Caswell wrote:
>>
>> > > 2. Can we do something with a bunch of hard-linked non-extendable
>> lists of
>> > > internal NIDs?
>> >
>> > > For example, providing GOST algorithms always requires a patch to
>> extend 3-5
>> > > internal lists.
>> > > If it could be done dynamically, it will be great.
>>
>> The simplest solution is to submit a PR to add your OIDs to OpenSSL,
>> so that no furher out of tree patches are required.
>>
>
> This is a way we go here and now. It is inevitable for libssl, but can be
> significantly reduced for libcrypto.
> Some examples are available in my response to Richard.
>
> And here we get a second problem, relatively small. If I remember
> correctly,
> adding new OIDs/NIDs is treated as breaking the binary compatibility so we
> have to wait for a major release.
>
>
>> Dynamic NIDs don't fit very well into the design, because NIDs are
>> expected to be stable compile-time constants.  We could perhaps
>> reserve a range for "private-use", and "engines" could allocate new
>> NIDs in the private space at runtime.  The key question is whether
>> such NIDs are global or valid only if returned to the same engine
>> (provider, ...).  If not global, the allocation might be static
>> within the engine, and not require any locks.
>>
>
> Totally agree. OBJ_create() and similar functions exist, but do not solve
> our problems.
>
> --
> SY, Dmitry Belyavsky
>
>
>

-- 
SY, Dmitry Belyavsky

Re: [openssl-project] OpenSSL 3.0 and FIPS Update

[Dr Paul Dale]

I don’t think that that new OIDs or NIDs are considering breaking.  Changing 
existing ones definitely is, but that’s an entirely different proposition.


Pauli
-- 
Dr Paul Dale | Cryptographer | Network Security & Encryption 
Phone +61 7 3031 7217
Oracle Australia



> On 25 Feb 2019, at 5:02 pm, Dmitry Belyavsky  wrote:
> 
> 
> 
> On Sun, Feb 24, 2019 at 11:31 PM Viktor Dukhovni  > wrote:
> On Thu, Feb 21, 2019 at 04:20:53PM +, Matt Caswell wrote:
> 
> > > 2. Can we do something with a bunch of hard-linked non-extendable lists of
> > > internal NIDs?
> >
> > > For example, providing GOST algorithms always requires a patch to extend 
> > > 3-5
> > > internal lists.
> > > If it could be done dynamically, it will be great.
> 
> The simplest solution is to submit a PR to add your OIDs to OpenSSL,
> so that no furher out of tree patches are required.
> 
> This is a way we go here and now. It is inevitable for libssl, but can be 
> significantly reduced for libcrypto.
> Some examples are available in my response to Richard.
> 
> And here we get a second problem, relatively small. If I remember correctly, 
> adding new OIDs/NIDs is treated as breaking the binary compatibility so we 
> have to wait for a major release.
>  
> Dynamic NIDs don't fit very well into the design, because NIDs are
> expected to be stable compile-time constants.  We could perhaps
> reserve a range for "private-use", and "engines" could allocate new
> NIDs in the private space at runtime.  The key question is whether
> such NIDs are global or valid only if returned to the same engine
> (provider, ...).  If not global, the allocation might be static
> within the engine, and not require any locks.
> 
> Totally agree. OBJ_create() and similar functions exist, but do not solve our 
> problems. 
> 
> -- 
> SY, Dmitry Belyavsky

Re: [openssl-project] OpenSSL 3.0 and FIPS Update

[Dmitry Belyavsky]

Dear Michael,

On Mon, Feb 25, 2019 at 2:41 AM Michael Richardson  wrote:

>
> Not sure who Matt quoted, wrote:
> >> 2. Can we do something with a bunch of hard-linked non-extendable
> >> lists of internal NIDs?
> >>
> >> For example, providing GOST algorithms always requires a patch to
> >> extend 3-5
> >> internal lists.
> >> If it could be done dynamically, it will be great.
>
> Matt then wrote:
> > The simplest solution is to submit a PR to add your OIDs to OpenSSL,
> > so that no furher out of tree patches are required.
>
> Viktor Dukhovni  wrote:
> > Dynamic NIDs don't fit very well into the design, because NIDs are
> > expected to be stable compile-time constants.  We could perhaps
> > reserve a range for "private-use", and "engines" could allocate new
> > NIDs in the private space at runtime.  The key question is whether
> > such NIDs are global or valid only if returned to the same engine
> > (provider, ...).  If not global, the allocation might be static
> > within the engine, and not require any locks.
>
> Having stubbed my toe on some NID stuff, I must question exposting NIDs.
> ruby-openssl used them in a dumb way that meant that adding extensions by
> OID
> was broken until I removed some code.
>
> I think that the #define/enum of NIDs should be made internal-only,
> available as optimization to internal code only.
> Your question then becomes, "are engines internal users", and I'd like the
> answer to be no. I think that the openssl 3 changes suggest the same thing.
>

The engines are _mostly_ external users. But sometimes, providing new
algorithms,
there appear some parts that should go into the core part. And regulation
creates similar problems.

All other users can call OBJ_obj2nid() or OBJ_txt2nid() to get a NID,
> and we can figure out how to allocate things dynamically if this makes
> sense.  I don't know which APIs are currently NID-only.


AFAIK, no external API, but there are some cases when external API does not
cover all.


-- 
SY, Dmitry Belyavsky

Re: [openssl-project] OpenSSL 3.0 and FIPS Update

[Dmitry Belyavsky]

On Sun, Feb 24, 2019 at 11:31 PM Viktor Dukhovni 
wrote:

> On Thu, Feb 21, 2019 at 04:20:53PM +, Matt Caswell wrote:
>
> > > 2. Can we do something with a bunch of hard-linked non-extendable
> lists of
> > > internal NIDs?
> >
> > > For example, providing GOST algorithms always requires a patch to
> extend 3-5
> > > internal lists.
> > > If it could be done dynamically, it will be great.
>
> The simplest solution is to submit a PR to add your OIDs to OpenSSL,
> so that no furher out of tree patches are required.
>

This is a way we go here and now. It is inevitable for libssl, but can be
significantly reduced for libcrypto.
Some examples are available in my response to Richard.

And here we get a second problem, relatively small. If I remember
correctly,
adding new OIDs/NIDs is treated as breaking the binary compatibility so we
have to wait for a major release.


> Dynamic NIDs don't fit very well into the design, because NIDs are
> expected to be stable compile-time constants.  We could perhaps
> reserve a range for "private-use", and "engines" could allocate new
> NIDs in the private space at runtime.  The key question is whether
> such NIDs are global or valid only if returned to the same engine
> (provider, ...).  If not global, the allocation might be static
> within the engine, and not require any locks.
>

Totally agree. OBJ_create() and similar functions exist, but do not solve
our problems.

-- 
SY, Dmitry Belyavsky

Re: [openssl-project] OpenSSL 3.0 and FIPS Update

[Michael Richardson]

Not sure who Matt quoted, wrote:
>> 2. Can we do something with a bunch of hard-linked non-extendable
>> lists of internal NIDs?
>>
>> For example, providing GOST algorithms always requires a patch to
>> extend 3-5
>> internal lists.
>> If it could be done dynamically, it will be great.

Matt then wrote:
> The simplest solution is to submit a PR to add your OIDs to OpenSSL,
> so that no furher out of tree patches are required.

Viktor Dukhovni  wrote:
> Dynamic NIDs don't fit very well into the design, because NIDs are
> expected to be stable compile-time constants.  We could perhaps
> reserve a range for "private-use", and "engines" could allocate new
> NIDs in the private space at runtime.  The key question is whether
> such NIDs are global or valid only if returned to the same engine
> (provider, ...).  If not global, the allocation might be static
> within the engine, and not require any locks.

Having stubbed my toe on some NID stuff, I must question exposting NIDs.
ruby-openssl used them in a dumb way that meant that adding extensions by OID
was broken until I removed some code.

I think that the #define/enum of NIDs should be made internal-only,
available as optimization to internal code only.
Your question then becomes, "are engines internal users", and I'd like the
answer to be no. I think that the openssl 3 changes suggest the same thing.

All other users can call OBJ_obj2nid() or OBJ_txt2nid() to get a NID,
and we can figure out how to allocate things dynamically if this makes
sense.  I don't know which APIs are currently NID-only.

--
]   Never tell me the odds! | ipv6 mesh networks [
]   Michael Richardson, Sandelman Software Works|IoT architect   [
] m...@sandelman.ca  http://www.sandelman.ca/|   ruby on rails[




signature.asc
Description: PGP signature

Re: [openssl-project] OpenSSL 3.0 and FIPS Update

[Viktor Dukhovni]

On Thu, Feb 21, 2019 at 04:20:53PM +, Matt Caswell wrote:

> > 2. Can we do something with a bunch of hard-linked non-extendable lists of
> > internal NIDs?
>
> > For example, providing GOST algorithms always requires a patch to extend 3-5
> > internal lists.
> > If it could be done dynamically, it will be great.

The simplest solution is to submit a PR to add your OIDs to OpenSSL,
so that no furher out of tree patches are required.

Dynamic NIDs don't fit very well into the design, because NIDs are
expected to be stable compile-time constants.  We could perhaps
reserve a range for "private-use", and "engines" could allocate new
NIDs in the private space at runtime.  The key question is whether
such NIDs are global or valid only if returned to the same engine
(provider, ...).  If not global, the allocation might be static
within the engine, and not require any locks.

-- 
Viktor.

Re: [openssl-project] OpenSSL 3.0 and FIPS Update

[Dmitry Belyavsky]

Dear Richard,

On Sat, Feb 23, 2019 at 8:47 AM Richard Levitte  wrote:

> On Thu, 21 Feb 2019 17:20:53 +0100,
> Matt Caswell wrote:
> > On 21/02/2019 15:02, Dmitry Belyavsky wrote:
> > > Dear Matt
> > >
> > >
> > >
> > > On Wed, Feb 13, 2019 at 9:30 PM Matt Caswell  > > > wrote:
> > >
> > > Please see my blog post for an OpenSSL 3.0 and FIPS Update:
> > >
> > > https://www.openssl.org/blog/blog/2019/02/13/FIPS-update/
> > >
> > >
> > > After reading the proposed architecture description, I have some
> questions that
> > > are very important for the developers of non-US certified
> openssl-based products.
> >
> > Hi Dmitry,
> >
> > Answers inserted.
> >
> > >
> > > 1. Will it still be available to implement custom RAND_methods via the
> new
> > > providers API?
> >
> > Yes, I expect this to be possible.
>
> This is something I'd like to see explored further.  OpenSSL 3.0 will
> target the EVP API primarly, and while we do talk about entropy with
> regards to FIPS, I haven't quite grasped if that would be a provider
> internal thing or if entropy is supposed to come from "elsewhere".
>
> Since our RAND API is separate from the EVP API, I'm unsure how we
> plan on getting custom RAND_methods from providers.
>
> Please note that we can add RAND to the list of provider backed APIs,
> and given a foundation that we're currently building, it may even be
> quite easy.  However, no one has said explicitly that we would do so.
>
> The other option is, of course, to move the RAND API to EVP somehow,
> but that will probably be more challenging.
>

I do not think it is really necessary to move RAND to EVP.
Current architecture suits our requirements, but if the possibility to
overwrite
the RAND_METHOD is removed, it will cause problems for us.


> > > 2. Can we do something with a bunch of hard-linked non-extendable
> lists of
> > > internal NIDs?
> > > For example, providing GOST algorithms always requires a patch to
> extend 3-5
> > > internal lists.
> > > If it could be done dynamically, it will be great.
> >
> > That's not currently something we've considered, but I agree it
> > would be great to fix that. Perhaps you could create a github issue
> > identifying the specific areas we should be looking at and then we
> > can take a look at the feasibility of fixing it.
>
> Let me address this in a different way...
>
> Are you very attached to those NIDs and them actually being NIDs?  Or
> would you be just as happy to have the implementations identified by
> name?  You see, providers will offer algorithm implementation by
> algorithm name (oh, and properties), not by number.
>

The command
grep -ril gost . | grep -v objects
in the crypto/ folder enlists the following files:

./cms/cms_sd.c
./asn1/asn_mime.c
./x509/x509type.c
./pkcs12/p12_mutl.c
./evp/evp_pbe.c
./pkcs7/pk7_smime.c

Namely the functions CMS_add_standard_smimecap, PKCS7_sign_add_signer,
asn1_write_micalg, X509_certificate_type and array builtin_pbe[] refer to
gost-related NIDs.

The pkcs12_gen_mac function has a gost-specific processing.
It was much more simple to add gost-specific processing here than to add
a callback everywhere, though it breaks encapsulation I dream about.

Also, we have some patches adding Russian-specific X.509 extensions,
and I think for now it's better to register the necessary NIDs and provide
pull requests to add their processing.

The situation in libssl is much more difficult, because of more monolithic
architecture there.


>
> > > 3. Do you have plans to make some callback structures created by
> providers?
> > > I mean such structures as SSL key exchange/authentication methods,
> X.509
> > > extensions etc.
> >
> > There aren't any plans to do that at the moment. There's nothing in the
> provider
> > design that would prevent us from doing so at some point in the future.
>

-- 
SY, Dmitry Belyavsky

Re: [openssl-project] OpenSSL 3.0 and FIPS Update

[Richard Levitte]

On Thu, 21 Feb 2019 17:20:53 +0100,
Matt Caswell wrote:
> On 21/02/2019 15:02, Dmitry Belyavsky wrote:
> > Dear Matt
> > 
> > 
> > 
> > On Wed, Feb 13, 2019 at 9:30 PM Matt Caswell  > > wrote:
> > 
> > Please see my blog post for an OpenSSL 3.0 and FIPS Update:
> > 
> > https://www.openssl.org/blog/blog/2019/02/13/FIPS-update/
> > 
> > 
> > After reading the proposed architecture description, I have some questions 
> > that
> > are very important for the developers of non-US certified openssl-based 
> > products.
> 
> Hi Dmitry,
> 
> Answers inserted.
> 
> > 
> > 1. Will it still be available to implement custom RAND_methods via the new
> > providers API?
> 
> Yes, I expect this to be possible.

This is something I'd like to see explored further.  OpenSSL 3.0 will
target the EVP API primarly, and while we do talk about entropy with
regards to FIPS, I haven't quite grasped if that would be a provider
internal thing or if entropy is supposed to come from "elsewhere".

Since our RAND API is separate from the EVP API, I'm unsure how we
plan on getting custom RAND_methods from providers.

Please note that we can add RAND to the list of provider backed APIs,
and given a foundation that we're currently building, it may even be
quite easy.  However, no one has said explicitly that we would do so.

The other option is, of course, to move the RAND API to EVP somehow,
but that will probably be more challenging.

> > 2. Can we do something with a bunch of hard-linked non-extendable lists of
> > internal NIDs? 
> > For example, providing GOST algorithms always requires a patch to extend 3-5
> > internal lists.
> > If it could be done dynamically, it will be great.
> 
> That's not currently something we've considered, but I agree it
> would be great to fix that. Perhaps you could create a github issue
> identifying the specific areas we should be looking at and then we
> can take a look at the feasibility of fixing it.

Let me address this in a different way...

Are you very attached to those NIDs and them actually being NIDs?  Or
would you be just as happy to have the implementations identified by
name?  You see, providers will offer algorithm implementation by
algorithm name (oh, and properties), not by number.

> > 3. Do you have plans to make some callback structures created by providers? 
> > I mean such structures as SSL key exchange/authentication methods, X.509
> > extensions etc.
> 
> There aren't any plans to do that at the moment. There's nothing in the 
> provider
> design that would prevent us from doing so at some point in the future.

Cheers,
Richard

-- 
Richard Levitte levi...@openssl.org
OpenSSL Project http://www.openssl.org/~levitte/

Re: [openssl-project] OpenSSL 3.0 and FIPS Update

[Matt Caswell]

On 21/02/2019 15:02, Dmitry Belyavsky wrote:
> Dear Matt
> 
> 
> 
> On Wed, Feb 13, 2019 at 9:30 PM Matt Caswell  > wrote:
> 
> Please see my blog post for an OpenSSL 3.0 and FIPS Update:
> 
> https://www.openssl.org/blog/blog/2019/02/13/FIPS-update/
> 
> 
> After reading the proposed architecture description, I have some questions 
> that
> are very important for the developers of non-US certified openssl-based 
> products.

Hi Dmitry,

Answers inserted.

> 
> 1. Will it still be available to implement custom RAND_methods via the new
> providers API?

Yes, I expect this to be possible.

> 2. Can we do something with a bunch of hard-linked non-extendable lists of
> internal NIDs? 
> For example, providing GOST algorithms always requires a patch to extend 3-5
> internal lists.
> If it could be done dynamically, it will be great.

That's not currently something we've considered, but I agree it would be great
to fix that. Perhaps you could create a github issue identifying the specific
areas we should be looking at and then we can take a look at the feasibility of
fixing it.

> 3. Do you have plans to make some callback structures created by providers? 
> I mean such structures as SSL key exchange/authentication methods, X.509
> extensions etc.

There aren't any plans to do that at the moment. There's nothing in the provider
design that would prevent us from doing so at some point in the future.

Matt

Re: [openssl-project] OpenSSL 3.0 and FIPS Update

[Dmitry Belyavsky]

Dear Matt



On Wed, Feb 13, 2019 at 9:30 PM Matt Caswell  wrote:

> Please see my blog post for an OpenSSL 3.0 and FIPS Update:
>
> https://www.openssl.org/blog/blog/2019/02/13/FIPS-update/


After reading the proposed architecture description, I have some questions
that are very important for the developers of non-US certified
openssl-based products.

1. Will it still be available to implement custom RAND_methods via the new
providers API?
2. Can we do something with a bunch of hard-linked non-extendable lists of
internal NIDs?
For example, providing GOST algorithms always requires a patch to extend
3-5 internal lists.
If it could be done dynamically, it will be great.
3. Do you have plans to make some callback structures created by providers?
I mean such structures as SSL key exchange/authentication methods, X.509
extensions etc.

Thank you very much!

-- 
SY, Dmitry Belyavsky

Re: [openssl-project] OpenSSL 3.0 and FIPS Update

[Matt Caswell]

On 15/02/2019 19:03, Sam Roberts wrote:
> I don't see a FIPS repo in https://github.com/openssl, or a FIPS
> branch in https://github.com/openssl/openssl/branches/all
>> Has coding started? If so, is it visible anywhere? If not, where
> should we watch for when it does?

All coding will be taking place in the master branch. The 3.0.0 release will
bring the FIPS module into mainline OpenSSL.

> 
> The FIPS design doc looks like lots of thought has gone into it, which
> is very promising.
> 
> I also looked around in github.com/openssl,  even the
> OpenSSL_1_0_2-stable branch, and couldn't find where the
> openssl-fips-2.0.16.tar.gz is built from. Where is it located?
You can checkout the OpenSSL-fips-2_0_16 tag, which is also on the
OpenSSL-fips-2_0-stable branch.

Matt

Re: [openssl-project] OpenSSL 3.0 and FIPS Update

[Richard Levitte]

On Fri, 15 Feb 2019 20:03:33 +0100,
Sam Roberts wrote:
> 
> I don't see a FIPS repo in https://github.com/openssl, or a FIPS
> branch in https://github.com/openssl/openssl/branches/all
> 
> Has coding started? If so, is it visible anywhere? If not, where
> should we watch for when it does?

Coding has started to appear on github since the beginning of this
week, and there's a related github project that we should attach
related issue and PRs to:

https://github.com/openssl/openssl/projects/2

That project should hold a collected view of everything that happens
when it does.

As for the FIPS module itself, it will not appear immediately.  We
need to code the foundation, i.e. the new framework, first.

> The FIPS design doc looks like lots of thought has gone into it,
> which is very promising.
> 
> I also looked around in github.com/openssl,  even the
> OpenSSL_1_0_2-stable branch, and couldn't find where the
> openssl-fips-2.0.16.tar.gz is built from. Where is it located?

There are branches called OpenSSL-fips-*, that's where you want to
look.  We will NOT use that as a model for the 3.0.0 FIPS module,
though.

Cheers,
Richard

-- 
Richard Levitte levi...@openssl.org
OpenSSL Project http://www.openssl.org/~levitte/

Re: [openssl-project] OpenSSL 3.0 and FIPS Update

[Sam Roberts]

I don't see a FIPS repo in https://github.com/openssl, or a FIPS
branch in https://github.com/openssl/openssl/branches/all

Has coding started? If so, is it visible anywhere? If not, where
should we watch for when it does?

The FIPS design doc looks like lots of thought has gone into it, which
is very promising.

I also looked around in github.com/openssl,  even the
OpenSSL_1_0_2-stable branch, and couldn't find where the
openssl-fips-2.0.16.tar.gz is built from. Where is it located?

Thanks,
Sam