Re: [openssl-dev] [RFC v2 2/2] pem: load engine keys

[Richard Levitte]

In message <584d7f4e.8090...@roumenpetrov.info> on Sun, 11 Dec 2016 18:31:10 
+0200, Roumen Petrov  said:

openssl> One remark for store load function api - in most cases (load from
openssl> file) it is password callback but is other cases it could be PIN or
openssl> something different.
openssl> Please use more generic description.
openssl> For instance engine callback is defined in generic way - ui_method and
openssl> its callback_data.

I see what you mean.  Just did the improvement.

-- 
Richard Levitte levi...@openssl.org
OpenSSL Project http://www.openssl.org/~levitte/
-- 
openssl-dev mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev

Re: [openssl-dev] [RFC v2 2/2] pem: load engine keys

[Richard Levitte]

Roumen Petrov  skrev: (11 december 2016 17:31:10 CET)
>Hi Richard,
>
>Richard Levitte wrote:
>> In message<20161206.223057.237264374331072901.levi...@openssl.org> 
>on Tue, 06 Dec 2016 22:30:57 +0100 (CET), Richard
>Levitte  said:
>>
>> levitte> [SNIP]
>>
>> The easiest was actually to rewrite PEM_read_bio_PrivateKey()
>> entirely, so it solely uses the internal store_file functions I've
>> provided.
>> I wonder what kind of impact this would have on the community at
>> large.
>
>PEM_read_bio_PrivateKey use custom password callback . You propose
>"Store-API" with UI_METHOD  as password callback.
>Rewrite of pem_read... method obsoletes pem_password_cb.
>
>What about to ensure a transition period?
>For instance in openssl 1.1 to provide new functions based on UI_METHOD
>and to mark existing as deprecated.
>
>
>One remark for store load function api - in most cases (load from file)
>it is password callback but is other cases it could be PIN or something
>different.
>Please use more generic description.
>For instance engine callback is defined in generic way -  ui_method and
>its callback_data.

Earlier, I mentioned an experimental branch, 
https://github.com/levitte/openssl/tree/tpm_engine-support?files=1

If you have a look, you'll find an added UI utility function to wrap a pem 
password callback in a UI_METHOD. 

>
> 
>
>> Cheers,
>> Richard
>
>Regards,
>Roumen

-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.
-- 
openssl-dev mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev

Re: [openssl-dev] [RFC v2 2/2] pem: load engine keys

[Roumen Petrov]

Hi Richard,

Richard Levitte wrote:

In message<20161206.223057.237264374331072901.levi...@openssl.org>  on Tue, 06 Dec 
2016 22:30:57 +0100 (CET), Richard Levitte  said:

levitte> [SNIP]

The easiest was actually to rewrite PEM_read_bio_PrivateKey()
entirely, so it solely uses the internal store_file functions I've
provided.
I wonder what kind of impact this would have on the community at
large.


PEM_read_bio_PrivateKey use custom password callback . You propose "Store-API" 
with UI_METHOD  as password callback.
Rewrite of pem_read... method obsoletes pem_password_cb.

What about to ensure a transition period?
For instance in openssl 1.1 to provide new functions based on UI_METHOD and to 
mark existing as deprecated.


One remark for store load function api - in most cases (load from file) it is 
password callback but is other cases it could be PIN or something different.
Please use more generic description.
For instance engine callback is defined in generic way -  ui_method and its 
callback_data.




Cheers,
Richard


Regards,
Roumen
--
openssl-dev mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev

Re: [openssl-dev] [RFC v2 2/2] pem: load engine keys

[Roumen Petrov]

HI Richard,

Richard Levitte wrote:

In message<58472e4f.3010...@roumenpetrov.info>  on Tue, 06 Dec 2016 23:31:59 +0200, 
Roumen Petrov  said:

openssl> Hi Richard,
openssl>
[SNIP]
openssl> > Check.  My STORE branch is made to support that.
openssl> One URI could represent more then one item.
openssl> STORE_INFO_types is enumerate but URI could be associated to custom
openssl> data (handle) and this data could be used to get other data(handles).
openssl>
openssl> See capi engine CAPI_KEY *capi_find_key(CAPI_CTX * ctx, const char
openssl> *id)
[SNIP]
openssl> Is above case PKEY is loaded only if CERT is located(found).

I'm trying to understand but am failing.  Looking at your example,
it's quite clear that what you want to retrieve is a key, even though
you have to go through the corresponding certificate to get to it.
After first review of API delared in openssl/store.h I misunderstand 
goal of load method.


I think that code of capi engine could be considered as sample what is 
need for an loadable module (engine) to use "OpenSSL Store API". I post 
above code just to get idea where currently is used an "external store api".
Just imagine how existing capi code could be changed to use store-API 
and to implement loader(scheme?).


I'm asking as currently there is no interface (API) that could associate 
key (private) and X.509 certificate. Currently engines implement custom 
command as work-around. For instance LOAD_CERT_CTRL (pkcs11 and e_nss) 
and LOAD_CERT_EVP(e_nss).


This one of areas where applications could benefit from "Store API".

I post a sniped from CAPI code because it is part of OpenSSL, but king 
of "external store api" is used by other engines.




However,*nothing*  stops anyone from making a loader for the "capi"
scheme (if there is such a thing) that has a load method that will
return the certificate (STORE_INFO_CERT) on the first call and the
associated key (STORE_INFO_PKEY) on the second for the same URI.  It's
all about caching information, and there is a context variable (type
STORE_LOADER_CTX, which is just a template type for loader defined
'struct store_loader_ctx_st') to be used exactly for that kind of
purpose.


I guess that "load" method is supposed to return all data at once.

Actually it is an iterator!

Please update comments before method and if possible to change name of 
method.




[SNIP]
In your example above, I fail to see where the custom data would be
needed...  And frankly, STORE is first of all meant to handle types
that can be used with the rest of OpenSSL.  That being said, adding a
"whatever" STORE_INFO type isn't very hard either.  I'm just not
terribly convinced yet, but let's keep talking, I'll probably
understand sooner or later what you're actually after.
I also fail to see why a store scheme has to return "custom data". Note 
that thread start from request for load TPM keys and some one mention 
that TMP key has custom data.



In addition to load of key from file there is one another interfaces 
where could be used store api - see RT4681.
The goal is expired from fact that X.509 load method is hidden in 
OpenSSL 1.1.
I guess that "by dir" and "by file" could be updated to use store api. 
Also applications has to able to register that a "store scheme" could by 
used by X.509 lookups.




Cheers,
Richard ( oh, and if example code is needed, I can provide )


+4 for OpenSSL store api ;)

Regards,
Roumen

--
openssl-dev mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev

Re: [openssl-dev] [RFC v2 2/2] pem: load engine keys

[James Bottomley]

On Fri, 2016-12-09 at 00:25 +, David Woodhouse wrote:
> On Thu, 2016-12-08 at 16:22 -0800, James Bottomley wrote:
> > 
> > I'm guessing you mean this:
> > 
> > https://www.trustedcomputinggroup.org/wp-content/uploads/TSS_Versio
> > n_1.2_Level_1_FINAL.pdf
> > 
> > ?  It still doesn't tell you who the expected parent of the key 
> > would be, which is the problem I'm currently trying to solve.
> 
> Right, in S3.23 ("Portable Data") or thereabouts. And yes, it doesn't
> indicate the parent key. As discussed, the TssBlob structure there is
> fairly pointless which is presumably why the TPM ENGINE didn't use 
> it, and just dumped the actual blob (which I later put into an
> OCTET_STRING  in the PEM files, IIRC).
> 
> I was hoping that the TPM 2.0 spec would fix that, maybe?

Not far enough through it yet to be definitive, but I don't think so
... there's even more parent like seeds in it.

James


smime.p7s
Description: S/MIME cryptographic signature
-- 
openssl-dev mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev

Re: [openssl-dev] [RFC v2 2/2] pem: load engine keys

[David Woodhouse]

On Thu, 2016-12-08 at 16:22 -0800, James Bottomley wrote:
> 
> I'm guessing you mean this:
> 
> https://www.trustedcomputinggroup.org/wp-content/uploads/TSS_Version_1.2_Level_1_FINAL.pdf
> 
> ?  It still doesn't tell you who the expected parent of the key would
> be, which is the problem I'm currently trying to solve.

Right, in S3.23 ("Portable Data") or thereabouts. And yes, it doesn't
indicate the parent key. As discussed, the TssBlob structure there is
fairly pointless which is presumably why the TPM ENGINE didn't use it,
and just dumped the actual blob (which I later put into an OCTET_STRING
 in the PEM files, IIRC).

I was hoping that the TPM 2.0 spec would fix that, maybe?

-- 
dwmw2

smime.p7s
Description: S/MIME cryptographic signature
-- 
openssl-dev mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev

Re: [openssl-dev] [RFC v2 2/2] pem: load engine keys

[James Bottomley]

On Thu, 2016-12-08 at 15:56 -0800, James Bottomley wrote:
> On Thu, 2016-12-08 at 23:44 +, David Woodhouse wrote:
> > On Tue, 2016-12-06 at 22:30 +0100, Richard Levitte wrote:
> > > Oh
> > > 
> > > I think I aired some thoughts on using PEM headers a very long
> > > while
> > > ago, but that never came into fruition, among others because I
> > > ended
> > > up doubting that it would be the best way in the long run.
> > > 
> > > These days, the use of PEM headers is considered old and kinda
> > > sorta
> > > deprecated, even though OpenSSL still produces encrypted private
> > > key
> > > PEM files that uses headers for the encryption metadata.  It
> > > seems
> > > that PKCS#8 is prefered "out there".
> > > 
> > > So I have to wonder, is PEM really the right way to go for this?
> > > Would it be just as possible to wrap a TSS key with a PKCS#8
> > > container, and use the associated attributes for the external
> > > data?
> > > Just a thought, though...  I can't do more than throw around
> > > ideas,
> > > considering how little I know about TPM.
> > 
> > I would definitely suggest that we *don't* want to do it with PEM
> > headers. Just put the additional information into the binary ASN.1
> > structure.
> 
> Which evil is lesser?  If we put it in ASN.1 we'll be defining our
> own
> instead of using the TSS defined one.  If we use headers, we can put
> the extra data in them and use the TSS defined ASN.1 for the key
> blob.
> 
> > The 2.0 version of the TssBlob (from §3.23 of the 1.2 spec) should
> > hopefully contain all the auxiliary information we need, without 
> > having to stick it in PEM headers.
> 
> Which of the many specs is this?

I'm guessing you mean this:

https://www.trustedcomputinggroup.org/wp-content/uploads/TSS_Version_1.2_Level_1_FINAL.pdf

?  It still doesn't tell you who the expected parent of the key would
be, which is the problem I'm currently trying to solve.

James



smime.p7s
Description: S/MIME cryptographic signature
-- 
openssl-dev mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev

Re: [openssl-dev] [RFC v2 2/2] pem: load engine keys

[James Bottomley]

On Thu, 2016-12-08 at 23:44 +, David Woodhouse wrote:
> On Tue, 2016-12-06 at 22:30 +0100, Richard Levitte wrote:
> > Oh
> > 
> > I think I aired some thoughts on using PEM headers a very long
> > while
> > ago, but that never came into fruition, among others because I
> > ended
> > up doubting that it would be the best way in the long run.
> > 
> > These days, the use of PEM headers is considered old and kinda
> > sorta
> > deprecated, even though OpenSSL still produces encrypted private
> > key
> > PEM files that uses headers for the encryption metadata.  It seems
> > that PKCS#8 is prefered "out there".
> > 
> > So I have to wonder, is PEM really the right way to go for this?
> > Would it be just as possible to wrap a TSS key with a PKCS#8
> > container, and use the associated attributes for the external data?
> > Just a thought, though...  I can't do more than throw around ideas,
> > considering how little I know about TPM.
> 
> I would definitely suggest that we *don't* want to do it with PEM
> headers. Just put the additional information into the binary ASN.1
> structure.

Which evil is lesser?  If we put it in ASN.1 we'll be defining our own
instead of using the TSS defined one.  If we use headers, we can put
the extra data in them and use the TSS defined ASN.1 for the key blob.

> The 2.0 version of the TssBlob (from §3.23 of the 1.2 spec) should
> hopefully contain all the auxiliary information we need, without 
> having to stick it in PEM headers.

Which of the many specs is this?

James


smime.p7s
Description: S/MIME cryptographic signature
-- 
openssl-dev mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev

Re: [openssl-dev] [RFC v2 2/2] pem: load engine keys

[David Woodhouse]

On Tue, 2016-12-06 at 22:30 +0100, Richard Levitte wrote:
> Oh
> 
> I think I aired some thoughts on using PEM headers a very long while
> ago, but that never came into fruition, among others because I ended
> up doubting that it would be the best way in the long run.
> 
> These days, the use of PEM headers is considered old and kinda sorta
> deprecated, even though OpenSSL still produces encrypted private key
> PEM files that uses headers for the encryption metadata.  It seems
> that PKCS#8 is prefered "out there".
> 
> So I have to wonder, is PEM really the right way to go for this?
> Would it be just as possible to wrap a TSS key with a PKCS#8
> container, and use the associated attributes for the external data?
> Just a thought, though...  I can't do more than throw around ideas,
> considering how little I know about TPM.

I would definitely suggest that we *don't* want to do it with PEM
headers. Just put the additional information into the binary ASN.1
structure.

The 2.0 version of the TssBlob (from §3.23 of the 1.2 spec) should
hopefully contain all the auxiliary information we need, without having
to stick it in PEM headers.

-- 
dwmw2

smime.p7s
Description: S/MIME cryptographic signature
-- 
openssl-dev mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev

Re: [openssl-dev] [RFC v2 2/2] pem: load engine keys

[Benjamin Kaduk]

On 12/06/2016 10:42 PM, Richard Levitte wrote:
> The easiest was actually to rewrite PEM_read_bio_PrivateKey()
> entirely, so it solely uses the internal store_file functions I've
> provided.
> I wonder what kind of impact this would have on the community at
> large.
>

If you do that, please ensure that secure memory is used for the
temporary buffers involved in reading private key material (assuming
that some form of https://github.com/openssl/openssl/pull/1700 gets merged).

-Ben
-- 
openssl-dev mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev

Re: [openssl-dev] [RFC v2 2/2] pem: load engine keys

[Richard Levitte]

In message <20161206.223057.237264374331072901.levi...@openssl.org> on Tue, 06 
Dec 2016 22:30:57 +0100 (CET), Richard Levitte  said:

levitte> That being said, it should certainly be easy enough to change the
levitte> appropriate places to make sure headers are available as well, and I
levitte> have zero issues adding a header parameter to the try_decode
levitte> prototype and associated functions.

Done.

levitte> One thing I didn't think of earlier is that PEM_bytes_read_bio()
levitte> checks the pem name against a known set, *or* in the private key case,
levitte> that the pem name ends with " PRIVATE KEY" (which "TSS KEY BLOB" does
levitte> not), so some kind of refactoring is needed to accomodate the
levitte> store_file_load() call either way.
levitte> (quite frankly, I'm slowly realising that the STORE_FILE_HANDLER code
levitte> can replace quite a lot of the discovery code in the PEM module, so
levitte> refactoring could be in order either way)

The easiest was actually to rewrite PEM_read_bio_PrivateKey()
entirely, so it solely uses the internal store_file functions I've
provided.
I wonder what kind of impact this would have on the community at
large.

Cheers,
Richard

-- 
Richard Levitte levi...@openssl.org
OpenSSL Project http://www.openssl.org/~levitte/
-- 
openssl-dev mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev

Re: [openssl-dev] [RFC v2 2/2] pem: load engine keys

[Richard Levitte]

In message <58472e4f.3010...@roumenpetrov.info> on Tue, 06 Dec 2016 23:31:59 
+0200, Roumen Petrov  said:

openssl> Hi Richard,
openssl> 
openssl> Richard Levitte wrote:
openssl> > [SNIP]
openssl> > James.Bottomley> 1. We agreed that usability is greatly enhanced if
openssl> > openssl simply loads
openssl> > James.Bottomley> a key when presented with the file/uri etc. without
openssl> > the user having
openssl> > James.Bottomley>   to specify what the format of a key is
openssl> >
openssl> > Check.  My STORE branch is made to support that.
openssl> One URI could represent more then one item.
openssl> STORE_INFO_types is enumerate but URI could be associated to custom
openssl> data (handle) and this data could be used to get other data(handles).
openssl> 
openssl> See capi engine CAPI_KEY *capi_find_key(CAPI_CTX * ctx, const char
openssl> *id)
openssl> ..
openssl> hstore = capi_open_store(ctx, NULL);
openssl> if (!hstore)
openssl> return NULL;
openssl> cert = capi_find_cert(ctx, id, hstore);
openssl> if (cert) {
openssl> key = capi_get_cert_key(ctx, cert);
openssl> CertFreeCertificateContext(cert);
openssl> }
openssl> CertCloseStore(hstore, 0);
openssl> ..
openssl> Is above case PKEY is loaded only if CERT is located(found).

I'm trying to understand but am failing.  Looking at your example,
it's quite clear that what you want to retrieve is a key, even though
you have to go through the corresponding certificate to get to it.

However, *nothing* stops anyone from making a loader for the "capi"
scheme (if there is such a thing) that has a load method that will
return the certificate (STORE_INFO_CERT) on the first call and the
associated key (STORE_INFO_PKEY) on the second for the same URI.  It's
all about caching information, and there is a context variable (type
STORE_LOADER_CTX, which is just a template type for loader defined
'struct store_loader_ctx_st') to be used exactly for that kind of
purpose.

In your example above, I fail to see where the custom data would be
needed...  And frankly, STORE is first of all meant to handle types
that can be used with the rest of OpenSSL.  That being said, adding a
"whatever" STORE_INFO type isn't very hard either.  I'm just not
terribly convinced yet, but let's keep talking, I'll probably
understand sooner or later what you're actually after.

Cheers,
Richard ( oh, and if example code is needed, I can provide )

-- 
Richard Levitte levi...@openssl.org
OpenSSL Project http://www.openssl.org/~levitte/
-- 
openssl-dev mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev

Re: [openssl-dev] [RFC v2 2/2] pem: load engine keys

[Roumen Petrov]

Hi Richard,

Richard Levitte wrote:

[SNIP]
James.Bottomley>1. We agreed that usability is greatly enhanced if openssl 
simply loads
James.Bottomley>   a key when presented with the file/uri etc. without the 
user having
James.Bottomley>   to specify what the format of a key is

Check.  My STORE branch is made to support that.

One URI could represent more then one item.
STORE_INFO_types is enumerate but URI could be associated to custom data 
(handle) and this data could be used to get other data(handles).


See capi engine CAPI_KEY *capi_find_key(CAPI_CTX * ctx, const char *id)
..
hstore = capi_open_store(ctx, NULL);
if (!hstore)
return NULL;
cert = capi_find_cert(ctx, id, hstore);
if (cert) {
key = capi_get_cert_key(ctx, cert);
CertFreeCertificateContext(cert);
}
CertCloseStore(hstore, 0);
..
Is above case PKEY is loaded only if CERT is located(found).

I guess that STORE_INFO_types could be bit-mask and to support 
application specific data in addition.



[SNIP]
Roumen
--
openssl-dev mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev

Re: [openssl-dev] [RFC v2 2/2] pem: load engine keys

[Richard Levitte]

In message <1481043672.4406.22.ca...@hansenpartnership.com> on Tue, 06 Dec 2016 
09:01:12 -0800, James Bottomley  said:

James.Bottomley> On Tue, 2016-12-06 at 17:47 +0100, Richard Levitte wrote:
James.Bottomley> > In message <1481042048.4406.14.ca...@hansenpartnership.com> 
on Tue,
James.Bottomley> > 06 Dec 2016 08:34:08 -0800, James Bottomley <
James.Bottomley> > james.bottom...@hansenpartnership.com> said:
James.Bottomley> > 
James.Bottomley> > James.Bottomley> On Tue, 2016-12-06 at 15:12 +0100, Richard 
Levitte
James.Bottomley> > wrote:
James.Bottomley> > James.Bottomley> > In message <
James.Bottomley> > 1480697558.2410.33.ca...@hansenpartnership.com> on Fri,
James.Bottomley> > James.Bottomley> > 02 Dec 2016 08:52:38 -0800, James 
Bottomley <
James.Bottomley> > James.Bottomley> > james.bottom...@hansenpartnership.com> 
said:
James.Bottomley> > James.Bottomley> > 
James.Bottomley> > James.Bottomley> > When I made that argument, I hadn't 
thought and
James.Bottomley> > felt things through
James.Bottomley> > James.Bottomley> > entirely.  Truth be told, I'm feeling 
very uneasy
James.Bottomley> > handing over the
James.Bottomley> > James.Bottomley> > reading and parsing of the PEM file to an 
engine. 
James.Bottomley> > However, handing
James.Bottomley> > James.Bottomley> > over the decoded data and leaving it to 
the engine
James.Bottomley> > to do something
James.Bottomley> > James.Bottomley> > sensible with it, no issues at all.
James.Bottomley> > James.Bottomley> 
James.Bottomley> > James.Bottomley> OK, can I pick on this a bit (I'll look at 
the store
James.Bottomley> > stuff later and
James.Bottomley> > James.Bottomley> reply after I've played with it)?  What is 
it that
James.Bottomley> > you imagine handing
James.Bottomley> > James.Bottomley> to the engine?  Some type of buffer that 
contains
James.Bottomley> > the full PEM
James.Bottomley> > James.Bottomley> representation?
James.Bottomley> > 
James.Bottomley> > In my STORE branch, you will find this in
James.Bottomley> > include/openssl/store_file.h:
James.Bottomley> > 
James.Bottomley> > /*
James.Bottomley> >  * The try_decode function is called to check if the 
blob of data
James.Bottomley> > can
James.Bottomley> >  * be used by this handler, and if it can, decodes it 
into a
James.Bottomley> > supported
James.Bottomley> >  * OpenSSL and returns a STORE_INFO with the recorded 
data.
James.Bottomley> >  * Input:
James.Bottomley> >  *pem_name: If this blob comes from a PEM file, 
this
James.Bottomley> > holds
James.Bottomley> >  *  the PEM name.  If it comes from 
another type
James.Bottomley> > of
James.Bottomley> >  *  file, this is NULL.
James.Bottomley> >  *blob: The blob of data to match with what 
this
James.Bottomley> > handler
James.Bottomley> >  *  can use.
James.Bottomley> >  *len:  The length of the blob.
James.Bottomley> >  *handler_ctx:  For a handler marked repeatable, 
this pointer
James.Bottomley> > can
James.Bottomley> >  *  be used to create a context for the 
handler. 
James.Bottomley> > IT IS
James.Bottomley> >  *  THE HANDLER'S RESPONSIBILITY TO 
CREATE AND
James.Bottomley> > DESTROY
James.Bottomley> >  *  THIS CONTEXT APPROPRIATELY, i.e. 
create on
James.Bottomley> > first call
James.Bottomley> >  *  and destroy when about to return 
NULL.
James.Bottomley> >  *password_ui:  Application UI method for getting a 
password.
James.Bottomley> >  *password_ui_data:
James.Bottomley> >  *  Application data to be passed to 
password_ui
James.Bottomley> > when
James.Bottomley> >  *  it's called.
James.Bottomley> >  * Output:
James.Bottomley> >  *a STORE_INFO
James.Bottomley> >  */
James.Bottomley> > typedef STORE_INFO *(*STORE_FILE_try_decode_fn)(const 
char
James.Bottomley> > *pem_name,
James.Bottomley> > const 
unsigned
James.Bottomley> > char *blob,
James.Bottomley> > size_t 
len,
James.Bottomley> > void
James.Bottomley> > **handler_ctx,
James.Bottomley> > const 
UI_METHOD
James.Bottomley> > *password_ui,
James.Bottomley> > void
James.Bottomley> > *password_ui_data);
James.Bottomley> > 
James.Bottomley> > This is the central function that tries to see if it can 
decode
James.Bottomley> > whatever's thrown at it.  As you can see, it gets handed the 
PEM name
James.Bottomley> > if it's called as part of a PEM read.  The |blob| is the 
decoded *and
James.Bottomley> > decrypted* data.
James.Bottomley> > 
James.Bottomley> > James.Bottomley> The reason I chose 

Re: [openssl-dev] [RFC v2 2/2] pem: load engine keys

[Benjamin Kaduk]

On 12/06/2016 11:01 AM, James Bottomley wrote:
> The next problem is that this is slightly harder simply to insert into
> the PEM code.  The BIO parsing is done in PEM_bytes_read_bio() not
>  PEM_read_bio_PrivateKey().  The easy way to cope with this would be to
> move PEM parsing into the ENGINE_find_engine_load_key() function and
> then hand the name, header, blob to the engine.  The disadvantage of
> this is that we'll end up pulling the PEM apart twice: once in
> ENGINE_find_engine_load_key() and then again in PEM_bytes_read_bio(). 
>  Is this OK?  If it is I can code up a v3.  If not, I can think about
> how we might integrate this into PEM_bytes_read_bio().

FWIW, you may find the refactored version at
https://github.com/openssl/openssl/pull/1700 easier to reason about.  I
expect to have some time to update the pull request this week.

-Ben
-- 
openssl-dev mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev

Re: [openssl-dev] [RFC v2 2/2] pem: load engine keys

[James Bottomley]

On Tue, 2016-12-06 at 17:47 +0100, Richard Levitte wrote:
> In message <1481042048.4406.14.ca...@hansenpartnership.com> on Tue,
> 06 Dec 2016 08:34:08 -0800, James Bottomley <
> james.bottom...@hansenpartnership.com> said:
> 
> James.Bottomley> On Tue, 2016-12-06 at 15:12 +0100, Richard Levitte
> wrote:
> James.Bottomley> > In message <
> 1480697558.2410.33.ca...@hansenpartnership.com> on Fri,
> James.Bottomley> > 02 Dec 2016 08:52:38 -0800, James Bottomley <
> James.Bottomley> > james.bottom...@hansenpartnership.com> said:
> James.Bottomley> > 
> James.Bottomley> > When I made that argument, I hadn't thought and
> felt things through
> James.Bottomley> > entirely.  Truth be told, I'm feeling very uneasy
> handing over the
> James.Bottomley> > reading and parsing of the PEM file to an engine. 
> However, handing
> James.Bottomley> > over the decoded data and leaving it to the engine
> to do something
> James.Bottomley> > sensible with it, no issues at all.
> James.Bottomley> 
> James.Bottomley> OK, can I pick on this a bit (I'll look at the store
> stuff later and
> James.Bottomley> reply after I've played with it)?  What is it that
> you imagine handing
> James.Bottomley> to the engine?  Some type of buffer that contains
> the full PEM
> James.Bottomley> representation?
> 
> In my STORE branch, you will find this in
> include/openssl/store_file.h:
> 
> /*
>  * The try_decode function is called to check if the blob of data
> can
>  * be used by this handler, and if it can, decodes it into a
> supported
>  * OpenSSL and returns a STORE_INFO with the recorded data.
>  * Input:
>  *pem_name: If this blob comes from a PEM file, this
> holds
>  *  the PEM name.  If it comes from another type
> of
>  *  file, this is NULL.
>  *blob: The blob of data to match with what this
> handler
>  *  can use.
>  *len:  The length of the blob.
>  *handler_ctx:  For a handler marked repeatable, this pointer
> can
>  *  be used to create a context for the handler. 
> IT IS
>  *  THE HANDLER'S RESPONSIBILITY TO CREATE AND
> DESTROY
>  *  THIS CONTEXT APPROPRIATELY, i.e. create on
> first call
>  *  and destroy when about to return NULL.
>  *password_ui:  Application UI method for getting a password.
>  *password_ui_data:
>  *  Application data to be passed to password_ui
> when
>  *  it's called.
>  * Output:
>  *a STORE_INFO
>  */
> typedef STORE_INFO *(*STORE_FILE_try_decode_fn)(const char
> *pem_name,
> const unsigned
> char *blob,
> size_t len,
> void
> **handler_ctx,
> const UI_METHOD
> *password_ui,
> void
> *password_ui_data);
> 
> This is the central function that tries to see if it can decode
> whatever's thrown at it.  As you can see, it gets handed the PEM name
> if it's called as part of a PEM read.  The |blob| is the decoded *and
> decrypted* data.
> 
> James.Bottomley> The reason I chose a BIO is that it's the basic
> abstract data handler
> James.Bottomley> for openssl. I can hand a buffer to the engine,
> sure, but I'd need to
> James.Bottomley> transform it to a BIO again (because it would need
> PEM parsing and all
> James.Bottomley> the PEM parsers speak BIOs), so it feels suboptimal.
> 
> With a try_decode function, I really do not see why there's a need to
> deal with the BIO...

OK, I get what you want.  There are two problems with this.  The first
is easy: I'm eventually going to need the BIO header as well because
TPM2 keys are going to need to describe external things that aren't in
the key representation (and even TPM1 keys might need to say who their
parent key is).  However, this is fixable by simply expanding your
prototype above to have const char *pem_name and const char *pem_header
... is that OK?

The next problem is that this is slightly harder simply to insert into
the PEM code.  The BIO parsing is done in PEM_bytes_read_bio() not
 PEM_read_bio_PrivateKey().  The easy way to cope with this would be to
move PEM parsing into the ENGINE_find_engine_load_key() function and
then hand the name, header, blob to the engine.  The disadvantage of
this is that we'll end up pulling the PEM apart twice: once in
ENGINE_find_engine_load_key() and then again in PEM_bytes_read_bio(). 
 Is this OK?  If it is I can code up a v3.  If not, I can think about
how we might integrate this into PEM_bytes_read_bio().

James



-- 
openssl-dev mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev

Re: [openssl-dev] [RFC v2 2/2] pem: load engine keys

[Richard Levitte]

In message <1481042048.4406.14.ca...@hansenpartnership.com> on Tue, 06 Dec 2016 
08:34:08 -0800, James Bottomley  said:

James.Bottomley> On Tue, 2016-12-06 at 15:12 +0100, Richard Levitte wrote:
James.Bottomley> > In message <1480697558.2410.33.ca...@hansenpartnership.com> 
on Fri,
James.Bottomley> > 02 Dec 2016 08:52:38 -0800, James Bottomley <
James.Bottomley> > james.bottom...@hansenpartnership.com> said:
James.Bottomley> > 
James.Bottomley> > When I made that argument, I hadn't thought and felt things 
through
James.Bottomley> > entirely.  Truth be told, I'm feeling very uneasy handing 
over the
James.Bottomley> > reading and parsing of the PEM file to an engine.  However, 
handing
James.Bottomley> > over the decoded data and leaving it to the engine to do 
something
James.Bottomley> > sensible with it, no issues at all.
James.Bottomley> 
James.Bottomley> OK, can I pick on this a bit (I'll look at the store stuff 
later and
James.Bottomley> reply after I've played with it)?  What is it that you imagine 
handing
James.Bottomley> to the engine?  Some type of buffer that contains the full PEM
James.Bottomley> representation?

In my STORE branch, you will find this in include/openssl/store_file.h:

/*
 * The try_decode function is called to check if the blob of data can
 * be used by this handler, and if it can, decodes it into a supported
 * OpenSSL and returns a STORE_INFO with the recorded data.
 * Input:
 *pem_name: If this blob comes from a PEM file, this holds
 *  the PEM name.  If it comes from another type of
 *  file, this is NULL.
 *blob: The blob of data to match with what this handler
 *  can use.
 *len:  The length of the blob.
 *handler_ctx:  For a handler marked repeatable, this pointer can
 *  be used to create a context for the handler.  IT IS
 *  THE HANDLER'S RESPONSIBILITY TO CREATE AND DESTROY
 *  THIS CONTEXT APPROPRIATELY, i.e. create on first call
 *  and destroy when about to return NULL.
 *password_ui:  Application UI method for getting a password.
 *password_ui_data:
 *  Application data to be passed to password_ui when
 *  it's called.
 * Output:
 *a STORE_INFO
 */
typedef STORE_INFO *(*STORE_FILE_try_decode_fn)(const char *pem_name,
const unsigned char *blob,
size_t len,
void **handler_ctx,
const UI_METHOD 
*password_ui,
void *password_ui_data);

This is the central function that tries to see if it can decode
whatever's thrown at it.  As you can see, it gets handed the PEM name
if it's called as part of a PEM read.  The |blob| is the decoded *and
decrypted* data.

James.Bottomley> The reason I chose a BIO is that it's the basic abstract data 
handler
James.Bottomley> for openssl. I can hand a buffer to the engine, sure, but I'd 
need to
James.Bottomley> transform it to a BIO again (because it would need PEM parsing 
and all
James.Bottomley> the PEM parsers speak BIOs), so it feels suboptimal.

With a try_decode function, I really do not see why there's a need to
deal with the BIO...

-- 
Richard Levitte levi...@openssl.org
OpenSSL Project http://www.openssl.org/~levitte/
-- 
openssl-dev mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev

Re: [openssl-dev] [RFC v2 2/2] pem: load engine keys

[James Bottomley]

On Tue, 2016-12-06 at 15:12 +0100, Richard Levitte wrote:
> In message <1480697558.2410.33.ca...@hansenpartnership.com> on Fri,
> 02 Dec 2016 08:52:38 -0800, James Bottomley <
> james.bottom...@hansenpartnership.com> said:
> 
> James.Bottomley> On Thu, 2016-12-01 at 09:30 +0100, Richard Levitte
> wrote:
> James.Bottomley> > 
> James.Bottomley> > James Bottomley <
> james.bottom...@hansenpartnership.com> skrev: (1
> James.Bottomley> > december 2016 07:36:26 CET)
> James.Bottomley> > > On Thu, 2016-12-01 at 01:38 +0100, Richard
> Levitte wrote:
> James.Bottomley> > > > 
> James.Bottomley> > > > James Bottomley <
> james.bottom...@hansenpartnership.com> skrev: (1
> James.Bottomley> > > > december 2016 00:42:09 CET)
> James.Bottomley> [...]
> James.Bottomley> > > > > On Thu, 2016-12-01 at 00:22 +0100, Richard
> Levitte wrote:
> James.Bottomley> > > > > > Generally speaking, I am unsure about your
> solution. It seems
> James.Bottomley> > > > > > like hack to fit a specific case where
> something more general
> James.Bottomley> > > > > > could be of greater service to others as
> well.
> James.Bottomley> > > > > 
> James.Bottomley> > > > > Well, the more adaptable patch set was the
> previous one that 
> James.Bottomley> > > > > overloaded the meaning of key_id.  This one
> has a specific bio 
> James.Bottomley> > > > > mechanism for loading PEM files, so it only
> really works for 
> James.Bottomley> > > > > engines that have PEM representable unloaded
> keys (which, to be
> James.Bottomley> > > > > fair, is almost all of them, since even the
> USB crypto keys 
> James.Bottomley> > > > > have a wrapped format).
> James.Bottomley> > > > > 
> James.Bottomley> > > > > I've tried to make it as generic as
> possible, but I am 
> James.Bottomley> > > > > conditioned to think to my use case: TPM
> keys.  If you give an 
> James.Bottomley> > > > > example of another use case, it will help me
> see where it 
> James.Bottomley> > > > > should be more generic.
> James.Bottomley> > > > 
> James.Bottomley> > > > Among other things, I'd rather not encourage
> an API that 
> James.Bottomley> > > > inherently makes the engine<->libcrypto tie
> even tighter. Also, 
> James.Bottomley> > > > it puts a requirement on the engine to parse
> PEM, which is
> James.Bottomley> > > > unnecessary.
> James.Bottomley> 
> James.Bottomley> Actually, I missed this initially.  This is
> definitely not a
> James.Bottomley> requirement: engines that wish to parse PEM keys may
> do so, but an
> James.Bottomley> engine that doesn't simply doesn't implement the
> callback.  Only
> James.Bottomley> engines that are loaded from the configuration file
> actually get asked
> James.Bottomley> if they understand the key, and then only if they
> supply the callback,
> James.Bottomley> so this is a decision on which engines to is made by
> the admin or
> James.Bottomley> packager (and the engine builder, who decides
> whether to implement or
> James.Bottomley> not).
> 
> When I made that argument, I hadn't thought and felt things through
> entirely.  Truth be told, I'm feeling very uneasy handing over the
> reading and parsing of the PEM file to an engine.  However, handing
> over the decoded data and leaving it to the engine to do something
> sensible with it, no issues at all.

OK, can I pick on this a bit (I'll look at the store stuff later and
reply after I've played with it)?  What is it that you imagine handing
to the engine?  Some type of buffer that contains the full PEM
representation?

The reason I chose a BIO is that it's the basic abstract data handler
for openssl. I can hand a buffer to the engine, sure, but I'd need to
transform it to a BIO again (because it would need PEM parsing and all
the PEM parsers speak BIOs), so it feels suboptimal.

James


-- 
openssl-dev mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev

Re: [openssl-dev] [RFC v2 2/2] pem: load engine keys

[Richard Levitte]

In message <1480697558.2410.33.ca...@hansenpartnership.com> on Fri, 02 Dec 2016 
08:52:38 -0800, James Bottomley  said:

James.Bottomley> On Thu, 2016-12-01 at 09:30 +0100, Richard Levitte wrote:
James.Bottomley> > 
James.Bottomley> > James Bottomley  
skrev: (1
James.Bottomley> > december 2016 07:36:26 CET)
James.Bottomley> > > On Thu, 2016-12-01 at 01:38 +0100, Richard Levitte wrote:
James.Bottomley> > > > 
James.Bottomley> > > > James Bottomley  
skrev: (1
James.Bottomley> > > > december 2016 00:42:09 CET)
James.Bottomley> [...]
James.Bottomley> > > > > On Thu, 2016-12-01 at 00:22 +0100, Richard Levitte 
wrote:
James.Bottomley> > > > > > Generally speaking, I am unsure about your solution. 
It seems
James.Bottomley> > > > > > like hack to fit a specific case where something 
more general
James.Bottomley> > > > > > could be of greater service to others as well.
James.Bottomley> > > > > 
James.Bottomley> > > > > Well, the more adaptable patch set was the previous 
one that 
James.Bottomley> > > > > overloaded the meaning of key_id.  This one has a 
specific bio 
James.Bottomley> > > > > mechanism for loading PEM files, so it only really 
works for 
James.Bottomley> > > > > engines that have PEM representable unloaded keys 
(which, to be
James.Bottomley> > > > > fair, is almost all of them, since even the USB crypto 
keys 
James.Bottomley> > > > > have a wrapped format).
James.Bottomley> > > > > 
James.Bottomley> > > > > I've tried to make it as generic as possible, but I am 
James.Bottomley> > > > > conditioned to think to my use case: TPM keys.  If you 
give an 
James.Bottomley> > > > > example of another use case, it will help me see where 
it 
James.Bottomley> > > > > should be more generic.
James.Bottomley> > > > 
James.Bottomley> > > > Among other things, I'd rather not encourage an API that 
James.Bottomley> > > > inherently makes the engine<->libcrypto tie even 
tighter. Also, 
James.Bottomley> > > > it puts a requirement on the engine to parse PEM, which 
is
James.Bottomley> > > > unnecessary.
James.Bottomley> 
James.Bottomley> Actually, I missed this initially.  This is definitely not a
James.Bottomley> requirement: engines that wish to parse PEM keys may do so, 
but an
James.Bottomley> engine that doesn't simply doesn't implement the callback.  
Only
James.Bottomley> engines that are loaded from the configuration file actually 
get asked
James.Bottomley> if they understand the key, and then only if they supply the 
callback,
James.Bottomley> so this is a decision on which engines to is made by the admin 
or
James.Bottomley> packager (and the engine builder, who decides whether to 
implement or
James.Bottomley> not).

When I made that argument, I hadn't thought and felt things through
entirely.  Truth be told, I'm feeling very uneasy handing over the
reading and parsing of the PEM file to an engine.  However, handing
over the decoded data and leaving it to the engine to do something
sensible with it, no issues at all.

James.Bottomley> If I summarise the arguments about self identifying files from 
the v1
James.Bottomley> thread:
James.Bottomley> 
James.Bottomley>1. We agreed that usability is greatly enhanced if openssl 
simply loads
James.Bottomley>   a key when presented with the file/uri etc. without the 
user having
James.Bottomley>   to specify what the format of a key is

Check.  My STORE branch is made to support that.

James.Bottomley>2. For PEM files, we believe this is easy because the 
guards uniquely
James.Bottomley>   identify the file format, so whatever key the 
application needs, we
James.Bottomley>   can verify is contained in the thing presented

Check.

James.Bottomley>3. THere was more debate on the files that aren't fully self
James.Bottomley>   describing, like DER.  The argument was that the DER 
structure is
James.Bottomley>   usually unique enough, so we could do this, but there 
were
James.Bottomley>   reservations.

Yup...  The current TSS KEY BLOB is among those reservations ;-)

James.Bottomley>4. I thought there was agreement that we could move 
forwards with the
James.Bottomley>   PEM bit because it was uncontroversial

We're not quite on the same page regarding the exact implementation.
However, I think I have a solution...

James.Bottomley> The current PEM key loading code is already hooked for PKCS8, 
to make
James.Bottomley> the above a reality, we should hook it for pkcs12 as well.  
Once you
James.Bottomley> have the store code, I believe it should be hooked for this as 
well. 
James.Bottomley>  In this scenario, I don't quite see why it's not a hack to 
hook for
James.Bottomley> pkcs8 (and presumably 12) but it is a hack to hook for engines 
and
James.Bottomley> store.
James.Bottomley> 
James.Bottomley> If we agree on DER, the ideal world becomes all apps simply use
James.Bottomley> 

Re: [openssl-dev] [RFC v2 2/2] pem: load engine keys

[James Bottomley]

On Thu, 2016-12-01 at 09:30 +0100, Richard Levitte wrote:
> 
> James Bottomley  skrev: (1
> december 2016 07:36:26 CET)
> > On Thu, 2016-12-01 at 01:38 +0100, Richard Levitte wrote:
> > > 
> > > James Bottomley  skrev: (1
> > > december 2016 00:42:09 CET)
[...]
> > > > On Thu, 2016-12-01 at 00:22 +0100, Richard Levitte wrote:
> > > > > Generally speaking, I am unsure about your solution. It seems
> > > > > like hack to fit a specific case where something more general
> > > > > could be of greater service to others as well.
> > > > 
> > > > Well, the more adaptable patch set was the previous one that 
> > > > overloaded the meaning of key_id.  This one has a specific bio 
> > > > mechanism for loading PEM files, so it only really works for 
> > > > engines that have PEM representable unloaded keys (which, to be
> > > > fair, is almost all of them, since even the USB crypto keys 
> > > > have a wrapped format).
> > > > 
> > > > I've tried to make it as generic as possible, but I am 
> > > > conditioned to think to my use case: TPM keys.  If you give an 
> > > > example of another use case, it will help me see where it 
> > > > should be more generic.
> > > 
> > > Among other things, I'd rather not encourage an API that 
> > > inherently makes the engine<->libcrypto tie even tighter. Also, 
> > > it puts a requirement on the engine to parse PEM, which is
> > > unnecessary.

Actually, I missed this initially.  This is definitely not a
requirement: engines that wish to parse PEM keys may do so, but an
engine that doesn't simply doesn't implement the callback.  Only
engines that are loaded from the configuration file actually get asked
if they understand the key, and then only if they supply the callback,
so this is a decision on which engines to is made by the admin or
packager (and the engine builder, who decides whether to implement or
not).

> > > Also, we already have thoughts on loading keys by uri references,
> > > and there may be new ideas and formats coming in the future. All
> > > this is tied together and solving it one small hack at a time is
> > > sub-optimal in the long run.
> > > 
> > > I'll repeat myself again, please have a look at the STORE stuff 
> > > I'm working on. TPM files could very well serve as a first use
> > > case.
> > 
> > That's this new pull request:
> > 
> > https://github.com/openssl/openssl/pull/2011/files
> 
> Yes. 
> 
> > Just so I understand you, you mean register a store handler from 
> > the engine for the TPM BLOB PEM files so that a reference to the 
> > file via the store can use the PEM file as an EVP_PKEY?
> 
> Essentially, yes. 
> 
> > That will work, I'm just not clear from the current pull request 
> > how the store would be integrated with the existing PEM file
> > handler
> > ...
> > Will every application have to be modified to use the new store, or
> > will you hook it like I did for the engine keys?
> 
> Generally speaking, I was thinking that applications would move to
> use the STORE API. 
> 
> > The think I'm looking for is to have a TPM PEM key file just work 
> > in place of a standard RSA private key file.
> 
> I get your point. That could be med as a call after the 
> PEM_bytes_read_bio call. At that point, the necessary data for an 
> internal store loading are available. It's still a hack, but if you 
> must... 

If I summarise the arguments about self identifying files from the v1
thread:

   1. We agreed that usability is greatly enhanced if openssl simply loads
  a key when presented with the file/uri etc. without the user having
  to specify what the format of a key is
   2. For PEM files, we believe this is easy because the guards uniquely
  identify the file format, so whatever key the application needs, we
  can verify is contained in the thing presented
   3. THere was more debate on the files that aren't fully self
  describing, like DER.  The argument was that the DER structure is
  usually unique enough, so we could do this, but there were
  reservations.
   4. I thought there was agreement that we could move forwards with the
  PEM bit because it was uncontroversial

The current PEM key loading code is already hooked for PKCS8, to make
the above a reality, we should hook it for pkcs12 as well.  Once you
have the store code, I believe it should be hooked for this as well. 
 In this scenario, I don't quite see why it's not a hack to hook for
pkcs8 (and presumably 12) but it is a hack to hook for engines and
store.

If we agree on DER, the ideal world becomes all apps simply use
OPENSSL_load_privkey() and no-one needs to worry about format for key
loading because any reasonable format just works.

James

-- 
openssl-dev mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev

Re: [openssl-dev] [RFC v2 2/2] pem: load engine keys

[Richard Levitte]

James Bottomley  skrev: (1 december 2016 
07:36:26 CET)
>On Thu, 2016-12-01 at 01:38 +0100, Richard Levitte wrote:
>> 
>> James Bottomley  skrev: (1
>> december 2016 00:42:09 CET)
>> > On Thu, 2016-12-01 at 00:22 +0100, Richard Levitte wrote:
>> > > This patch doesn't fit the rest... 
>> > 
>> > I'm not quite sure I follow why.
>> 
>> It casts bp to const char *. That was for your earlier
>> implementation, wasn't it? It doesn't fit the latest incarnation of
>> your API. 
>
>Good catch: a leftover I missed the warning for in the spray of other
>compile information.  Apparently the debug-linux-x86_64 target doesn't
>have a -Werror ... I've added it so I should pick a problem like this
>up easily.

The debug config targets create binaries that can easily be run through a 
debugger, nothing more. What you're looking for is the configuration option 
--strict-warnings. 

>> > > Generally speaking, I am unsure about your solution. It seems 
>> > > like hack to fit a specific case where something more general 
>> > > could be of greater service to others as well.
>> > 
>> > Well, the more adaptable patch set was the previous one that 
>> > overloaded the meaning of key_id.  This one has a specific bio 
>> > mechanism for loading PEM files, so it only really works for 
>> > engines that have PEM representable unloaded keys (which, to be 
>> > fair, is almost all of them, since even the USB crypto keys have a
>> > wrapped format).
>> > 
>> > I've tried to make it as generic as possible, but I am conditioned 
>> > to think to my use case: TPM keys.  If you give an example of 
>> > another use case, it will help me see where it should be more
>> > generic.
>> 
>> Among other things, I'd rather not encourage an API that inherently
>> makes the engine<->libcrypto tie even tighter. Also, it puts a
>> requirement on the engine to parse PEM, which is unnecessary. 
>> 
>> Also, we already have thoughts on loading keys by uri references, and
>> there may be new ideas and formats coming in the future. All this is
>> tied together and solving it one small hack at a time is sub-optimal
>> in the long run. 
>> 
>> I'll repeat myself again, please have a look at the STORE stuff I'm
>> working on. TPM files could very well serve as a first use case. 
>
>That's this new pull request:
>
>https://github.com/openssl/openssl/pull/2011/files

Yes. 

>Just so I understand you, you mean register a store handler from the
>engine for the TPM BLOB PEM files so that a reference to the file via
>the store can use the PEM file as an EVP_PKEY?

Essentially, yes. 

>That will work, I'm just not clear from the current pull request how
>the store would be integrated with the existing PEM file handler ...
>Will every application have to be modified to use the new store, or
>will you hook it like I did for the engine keys?

Generally speaking, I was thinking that applications would move to use the 
STORE API. 

>The think I'm looking for is to have a TPM PEM key file just work in
>place of a standard RSA private key file.

I get your point. That could be med as a call after the PEM_bytes_read_bio 
call. At that point, the necessary data for an internal store loading are 
available. It's still a hack, but if you must... 

>
>James
>
>
>> > James
>> > 
>> > 
>> > > Cheers 
>> > > Richard 
>> > > 
>> > > On November 30, 2016 4:27:49 PM GMT+01:00, James Bottomley <
>> > > james.bottom...@hansenpartnership.com> wrote:
>> > > > Before trying to process the PEM file, hand it to each of the
>> > > > loaded
>> > > > engines to see if they recognise the PEM guards.  This uses the
>> > > > new
>> > > > bio based load key callback, so the engine must be loaded and
>> > > > implement this callback to be considered.
>> > > > 
>> > > > Signed-off-by: James Bottomley 
>> > > > ---
>> > > > crypto/pem/pem_pkey.c | 4 
>> > > > 1 file changed, 4 insertions(+)
>> > > > 
>> > > > diff --git a/crypto/pem/pem_pkey.c b/crypto/pem/pem_pkey.c
>> > > > index 04d6319..e3737f0 100644
>> > > > --- a/crypto/pem/pem_pkey.c
>> > > > +++ b/crypto/pem/pem_pkey.c
>> > > > @@ -85,6 +85,10 @@ EVP_PKEY *PEM_read_bio_PrivateKey(BIO *bp,
>> > > > EVP_PKEY
>> > > > **x, pem_password_cb *cb,
>> > > > int slen;
>> > > > EVP_PKEY *ret = NULL;
>> > > > 
>> > > > +/* first check to see if an engine can load the PEM */
>> > > > +if (ENGINE_find_engine_load_key(NULL, , (const char
>> > > > *)bp,
>> > > > cb,
>> > > > u) == 1)
>> > > > +return ret;
>> > > > +
>> > > > if (!PEM_bytes_read_bio(, , , PEM_STRING_EVP_PKEY,
>> > > > bp,
>> > > > cb,
>> > > > u))
>> > > > return NULL;
>> > > > p = data;
>> > > 
>> > > -- 
>> > > levi...@openssl.org 
>> 
>> -- 
>> Sent from my Android device with K-9 Mail. Please excuse my brevity.

-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.
-- 
openssl-dev mailing list
To unsubscribe: 

Re: [openssl-dev] [RFC v2 2/2] pem: load engine keys

[James Bottomley]

On Thu, 2016-12-01 at 01:38 +0100, Richard Levitte wrote:
> 
> James Bottomley  skrev: (1
> december 2016 00:42:09 CET)
> > On Thu, 2016-12-01 at 00:22 +0100, Richard Levitte wrote:
> > > This patch doesn't fit the rest... 
> > 
> > I'm not quite sure I follow why.
> 
> It casts bp to const char *. That was for your earlier
> implementation, wasn't it? It doesn't fit the latest incarnation of
> your API. 

Good catch: a leftover I missed the warning for in the spray of other
compile information.  Apparently the debug-linux-x86_64 target doesn't
have a -Werror ... I've added it so I should pick a problem like this
up easily.

> > > Generally speaking, I am unsure about your solution. It seems 
> > > like hack to fit a specific case where something more general 
> > > could be of greater service to others as well.
> > 
> > Well, the more adaptable patch set was the previous one that 
> > overloaded the meaning of key_id.  This one has a specific bio 
> > mechanism for loading PEM files, so it only really works for 
> > engines that have PEM representable unloaded keys (which, to be 
> > fair, is almost all of them, since even the USB crypto keys have a
> > wrapped format).
> > 
> > I've tried to make it as generic as possible, but I am conditioned 
> > to think to my use case: TPM keys.  If you give an example of 
> > another use case, it will help me see where it should be more
> > generic.
> 
> Among other things, I'd rather not encourage an API that inherently
> makes the engine<->libcrypto tie even tighter. Also, it puts a
> requirement on the engine to parse PEM, which is unnecessary. 
> 
> Also, we already have thoughts on loading keys by uri references, and
> there may be new ideas and formats coming in the future. All this is
> tied together and solving it one small hack at a time is sub-optimal
> in the long run. 
> 
> I'll repeat myself again, please have a look at the STORE stuff I'm
> working on. TPM files could very well serve as a first use case. 

That's this new pull request:

https://github.com/openssl/openssl/pull/2011/files

?

Just so I understand you, you mean register a store handler from the
engine for the TPM BLOB PEM files so that a reference to the file via
the store can use the PEM file as an EVP_PKEY?

That will work, I'm just not clear from the current pull request how
the store would be integrated with the existing PEM file handler ...
Will every application have to be modified to use the new store, or
will you hook it like I did for the engine keys?

The think I'm looking for is to have a TPM PEM key file just work in
place of a standard RSA private key file.

James


> > James
> > 
> > 
> > > Cheers 
> > > Richard 
> > > 
> > > On November 30, 2016 4:27:49 PM GMT+01:00, James Bottomley <
> > > james.bottom...@hansenpartnership.com> wrote:
> > > > Before trying to process the PEM file, hand it to each of the
> > > > loaded
> > > > engines to see if they recognise the PEM guards.  This uses the
> > > > new
> > > > bio based load key callback, so the engine must be loaded and
> > > > implement this callback to be considered.
> > > > 
> > > > Signed-off-by: James Bottomley 
> > > > ---
> > > > crypto/pem/pem_pkey.c | 4 
> > > > 1 file changed, 4 insertions(+)
> > > > 
> > > > diff --git a/crypto/pem/pem_pkey.c b/crypto/pem/pem_pkey.c
> > > > index 04d6319..e3737f0 100644
> > > > --- a/crypto/pem/pem_pkey.c
> > > > +++ b/crypto/pem/pem_pkey.c
> > > > @@ -85,6 +85,10 @@ EVP_PKEY *PEM_read_bio_PrivateKey(BIO *bp,
> > > > EVP_PKEY
> > > > **x, pem_password_cb *cb,
> > > > int slen;
> > > > EVP_PKEY *ret = NULL;
> > > > 
> > > > +/* first check to see if an engine can load the PEM */
> > > > +if (ENGINE_find_engine_load_key(NULL, , (const char
> > > > *)bp,
> > > > cb,
> > > > u) == 1)
> > > > +return ret;
> > > > +
> > > > if (!PEM_bytes_read_bio(, , , PEM_STRING_EVP_PKEY,
> > > > bp,
> > > > cb,
> > > > u))
> > > > return NULL;
> > > > p = data;
> > > 
> > > -- 
> > > levi...@openssl.org 
> 
> -- 
> Sent from my Android device with K-9 Mail. Please excuse my brevity.

-- 
openssl-dev mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev

Re: [openssl-dev] [RFC v2 2/2] pem: load engine keys

[Richard Levitte]

James Bottomley  skrev: (1 december 2016 
00:42:09 CET)
>On Thu, 2016-12-01 at 00:22 +0100, Richard Levitte wrote:
>> This patch doesn't fit the rest... 
>
>I'm not quite sure I follow why.

It casts bp to const char *. That was for your earlier implementation, wasn't 
it? It doesn't fit the latest incarnation of your API. 

>> Generally speaking, I am unsure about your solution. It seems like
>> hack to fit a specific case where something more general could be of
>> greater service to others as well. 
>
>Well, the more adaptable patch set was the previous one that overloaded
>the meaning of key_id.  This one has a specific bio mechanism for
>loading PEM files, so it only really works for engines that have PEM
>representable unloaded keys (which, to be fair, is almost all of them,
>since even the USB crypto keys have a wrapped format).
>
>I've tried to make it as generic as possible, but I am conditioned to
>think to my use case: TPM keys.  If you give an example of another use
>case, it will help me see where it should be more generic.

Among other things, I'd rather not encourage an API that inherently makes the 
engine<->libcrypto tie even tighter. Also, it puts a requirement on the engine 
to parse PEM, which is unnecessary. 

Also, we already have thoughts on loading keys by uri references, and there may 
be new ideas and formats coming in the future. All this is tied together and 
solving it one small hack at a time is sub-optimal in the long run. 

I'll repeat myself again, please have a look at the STORE stuff I'm working on. 
TPM files could very well serve as a first use case. 

>James
>
>
>> Cheers 
>> Richard 
>> 
>> On November 30, 2016 4:27:49 PM GMT+01:00, James Bottomley <
>> james.bottom...@hansenpartnership.com> wrote:
>> > Before trying to process the PEM file, hand it to each of the
>> > loaded
>> > engines to see if they recognise the PEM guards.  This uses the new
>> > bio based load key callback, so the engine must be loaded and
>> > implement this callback to be considered.
>> > 
>> > Signed-off-by: James Bottomley 
>> > ---
>> > crypto/pem/pem_pkey.c | 4 
>> > 1 file changed, 4 insertions(+)
>> > 
>> > diff --git a/crypto/pem/pem_pkey.c b/crypto/pem/pem_pkey.c
>> > index 04d6319..e3737f0 100644
>> > --- a/crypto/pem/pem_pkey.c
>> > +++ b/crypto/pem/pem_pkey.c
>> > @@ -85,6 +85,10 @@ EVP_PKEY *PEM_read_bio_PrivateKey(BIO *bp,
>> > EVP_PKEY
>> > **x, pem_password_cb *cb,
>> > int slen;
>> > EVP_PKEY *ret = NULL;
>> > 
>> > +/* first check to see if an engine can load the PEM */
>> > +if (ENGINE_find_engine_load_key(NULL, , (const char *)bp,
>> > cb,
>> > u) == 1)
>> > +return ret;
>> > +
>> > if (!PEM_bytes_read_bio(, , , PEM_STRING_EVP_PKEY, bp,
>> > cb,
>> > u))
>> > return NULL;
>> > p = data;
>> 
>> -- 
>> levi...@openssl.org 

-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.
-- 
openssl-dev mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev

Re: [openssl-dev] [RFC v2 2/2] pem: load engine keys

[James Bottomley]

On Thu, 2016-12-01 at 00:22 +0100, Richard Levitte wrote:
> This patch doesn't fit the rest... 

I'm not quite sure I follow why.  To allow engines to load PEM encoded
engine keys in place of machine processed ones, the hook into the
loader has to be in somewhere.  This seems to be the most generic place
to put the hook.

> Generally speaking, I am unsure about your solution. It seems like
> hack to fit a specific case where something more general could be of
> greater service to others as well. 

Well, the more adaptable patch set was the previous one that overloaded
the meaning of key_id.  This one has a specific bio mechanism for
loading PEM files, so it only really works for engines that have PEM
representable unloaded keys (which, to be fair, is almost all of them,
since even the USB crypto keys have a wrapped format).

I've tried to make it as generic as possible, but I am conditioned to
think to my use case: TPM keys.  If you give an example of another use
case, it will help me see where it should be more generic.

James


> Cheers 
> Richard 
> 
> On November 30, 2016 4:27:49 PM GMT+01:00, James Bottomley <
> james.bottom...@hansenpartnership.com> wrote:
> > Before trying to process the PEM file, hand it to each of the
> > loaded
> > engines to see if they recognise the PEM guards.  This uses the new
> > bio based load key callback, so the engine must be loaded and
> > implement this callback to be considered.
> > 
> > Signed-off-by: James Bottomley 
> > ---
> > crypto/pem/pem_pkey.c | 4 
> > 1 file changed, 4 insertions(+)
> > 
> > diff --git a/crypto/pem/pem_pkey.c b/crypto/pem/pem_pkey.c
> > index 04d6319..e3737f0 100644
> > --- a/crypto/pem/pem_pkey.c
> > +++ b/crypto/pem/pem_pkey.c
> > @@ -85,6 +85,10 @@ EVP_PKEY *PEM_read_bio_PrivateKey(BIO *bp,
> > EVP_PKEY
> > **x, pem_password_cb *cb,
> > int slen;
> > EVP_PKEY *ret = NULL;
> > 
> > +/* first check to see if an engine can load the PEM */
> > +if (ENGINE_find_engine_load_key(NULL, , (const char *)bp,
> > cb,
> > u) == 1)
> > +return ret;
> > +
> > if (!PEM_bytes_read_bio(, , , PEM_STRING_EVP_PKEY, bp,
> > cb,
> > u))
> > return NULL;
> > p = data;
> 
> -- 
> levi...@openssl.org 

-- 
openssl-dev mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev

Re: [openssl-dev] [RFC v2 2/2] pem: load engine keys

[Richard Levitte]

This patch doesn't fit the rest... 

Generally speaking, I am unsure about your solution. It seems like hack to fit 
a specific case where something more general could be of greater service to 
others as well. 

Cheers 
Richard 

On November 30, 2016 4:27:49 PM GMT+01:00, James Bottomley 
 wrote:
>Before trying to process the PEM file, hand it to each of the loaded
>engines to see if they recognise the PEM guards.  This uses the new
>bio based load key callback, so the engine must be loaded and
>implement this callback to be considered.
>
>Signed-off-by: James Bottomley 
>---
> crypto/pem/pem_pkey.c | 4 
> 1 file changed, 4 insertions(+)
>
>diff --git a/crypto/pem/pem_pkey.c b/crypto/pem/pem_pkey.c
>index 04d6319..e3737f0 100644
>--- a/crypto/pem/pem_pkey.c
>+++ b/crypto/pem/pem_pkey.c
>@@ -85,6 +85,10 @@ EVP_PKEY *PEM_read_bio_PrivateKey(BIO *bp, EVP_PKEY
>**x, pem_password_cb *cb,
> int slen;
> EVP_PKEY *ret = NULL;
> 
>+/* first check to see if an engine can load the PEM */
>+if (ENGINE_find_engine_load_key(NULL, , (const char *)bp, cb,
>u) == 1)
>+return ret;
>+
>if (!PEM_bytes_read_bio(, , , PEM_STRING_EVP_PKEY, bp, cb,
>u))
> return NULL;
> p = data;

-- 
levi...@openssl.org 
-- 
openssl-dev mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev

Re: [openssl-dev] [RFC v2 2/2] pem: load engine keys

[Salz, Rich]

> I suspect IBM will need to sign a CCLA ... they'll definitely need to know who
> your lawyers are.

We have a CCLA from IBM; contact Christopher Barrett.

> I did check those links ... they don't have any governance information about
> the actual openssl foundation that I can find.

If you want particular details, email the info address I gave you.  And again, 
that organization does not specify how the openssl project works.
-- 
openssl-dev mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev

Re: [openssl-dev] [RFC v2 2/2] pem: load engine keys

[James Bottomley]

On Wed, 2016-11-30 at 19:32 +, Salz, Rich wrote:
> > OK, so where is the foundation charter and who are your lawyers?
> 
> Wow, this seems to have taken a turn to the unfriendly.  I apologize
> if I added to that.  Sometimes a smiley doesn't wipe out all bad
> impressions.

No, it's standard if you insist on the CLA route:  If you sign a CLA to
an organization, you have to understand what the organization does
before you understand what you're actually committing to: the actual
commitment isn't within the four corners of the CLA.  Your current ICLA
gives effectively an unlimited perpetual licence to do anything with
regard to sublicensing so the scope of that grant is actually governed
by the bylaws and restrictions of the foundation itself because it is
the grant recipient.  For instance, the old OpenStack ICLA was fairly
similar to yours so you had to dig into their bylaws to understand what
they were actually committing to do with the code (basically sublicense
it to all comers under ASL-2).  The structure of the organisation
matters a lot: unlimited grant to a corporate entity under a CLA is
usually a bad idea because they often have nefarious plans to take your
code private via a dual licence, or they might mean well, but their
intentions become nullified if they get taken over.  Foundations are
usually better because their charter often restricts what they can
actually do with the code and what happens to the grant in the event of
dissolution or takeover, which is why reading and understanding the
charter (and possibly the bylaws) is important.

Usually I don't have to ask, all of this is simply available on the web
most of the time.  It's just the openssl foundation doesn't appear to
have any of this online.

I suspect IBM will need to sign a CCLA ... they'll definitely need to
know who your lawyers are.

> The OpenSSL Software Foundation is incorporated in the the state of
> Delaware, United States, as a non-profit corporation. It does not
> qualify as a tax-exempt charitable organisation under Section
> 501(c)(3) of the U.S. Internal Revenue Code.   You can email 
> i...@opensslfoundation.org with questions.
> But do note that openssl open source project itself is not governed
> by those entities, but rather by the collection of individuals known
> as the development team.  You can find more information by clicking
> on the "policies" and "community" tab on the website.  

I did check those links ... they don't have any governance information
about the actual openssl foundation that I can find.

James

-- 
openssl-dev mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev

Re: [openssl-dev] [RFC v2 2/2] pem: load engine keys

[Salz, Rich]

> OK, so where is the foundation charter and who are your lawyers?

Wow, this seems to have taken a turn to the unfriendly.  I apologize if I added 
to that.  Sometimes a smiley doesn't wipe out all bad impressions.

The OpenSSL Software Foundation is incorporated in the the state of Delaware, 
United States, as a non-profit corporation. It does not qualify as a tax-exempt 
charitable organisation under Section 501(c)(3) of the U.S. Internal Revenue 
Code.   You can email i...@opensslfoundation.org with questions.
But do note that openssl open source project itself is not governed by those 
entities, but rather by the collection of individuals known as the development 
team.  You can find more information by clicking on the "policies" and 
"community" tab on the website.  

Hope this helps.
-- 
openssl-dev mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev

Re: [openssl-dev] [RFC v2 2/2] pem: load engine keys

[James Bottomley]

On Wed, 2016-11-30 at 17:59 +, Salz, Rich wrote:
>  
> > Plus the DCO is industry best practice: even OpenStack is adopting
> > it after a
> > long struggle.
> 
> Great.  Good for them.
> 
> This is what we're doing.
> 
> :)

OK, so where is the foundation charter and who are your lawyers?

James

-- 
openssl-dev mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev

Re: [openssl-dev] [RFC v2 2/2] pem: load engine keys

[Salz, Rich]

 
> Plus the DCO is industry best practice: even OpenStack is adopting it after a
> long struggle.

Great.  Good for them.

This is what we're doing.

:)

-- 
openssl-dev mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev

Re: [openssl-dev] [RFC v2 2/2] pem: load engine keys

[James Bottomley]

On Wed, 2016-11-30 at 16:04 +, Salz, Rich wrote:
> > Groan ... since you're changing licences, I don't suppose you'd 
> > consider moving to a DCO model.
> 
> Sorry, no.  Legal advice and best practices.

Interesting: whose legal advice?  I assumed you were talking to the
SFLC and I thought they were mostly pre-programmed not to recommend
CLAs after their FSF experience.  I'm asking professionally because the
knee jerk reaction of most lawyers is to recommend a CLA and there's a
group programme (run by the LF) to help them kick the habit.

Plus the DCO is industry best practice: even OpenStack is adopting it
after a long struggle.

James

-- 
openssl-dev mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev

Re: [openssl-dev] [RFC v2 2/2] pem: load engine keys

[Salz, Rich]

> Actually, being a kernel developer, email is far easier.  I'll send a pull 
> request
> when everyone's OK with the mechanism, plus it will need tests and other
> things.

Well... okay.  I don't know how the community will react.  But I *do* know that 
the team prefers things as PR's.


> Groan ... since you're changing licences, I don't suppose you'd consider
> moving to a DCO model. 

Sorry, no.  Legal advice and best practices.
-- 
openssl-dev mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev

Re: [openssl-dev] [RFC v2 2/2] pem: load engine keys

[James Bottomley]

> Thanks for working to improve openssl.

You're welcome.

> It is probably easier for you to do a GitHub pull request and then
> have discussion here, pointing to that PR.

Actually, being a kernel developer, email is far easier.  I'll send a
pull request when everyone's OK with the mechanism, plus it will need
tests and other things.

> And also, before any of this code could be used, we'll need the
> appropriate CLA.

Groan ... since you're changing licences, I don't suppose you'd
consider moving to a DCO model.  It's a whole lot less bureaucratic and
much easier to keep track of.

James

-- 
openssl-dev mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev

Re: [openssl-dev] [RFC v2 2/2] pem: load engine keys

[Salz, Rich]

Thanks for working to improve openssl.

It is probably easier for you to do a GitHub pull request and then have 
discussion here, pointing to that PR.
And also, before any of this code could be used, we'll need the appropriate CLA.
-- 
openssl-dev mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev