ODP: CSR creation using pkcs11 dynamic engine

[Piotr Lobacz]

Ok i have written printf in src/p11_rsa.c to check if i'm entering the 
RSA_set_flags callback and yes i am. Printf is being displayed but i still got 
this error, which shouldn't be any more.

Od: openssl-users  w imieniu użytkownika 
Piotr Lobacz 
Wysłane: wtorek, 1 czerwca 2021 17:36
Do: Selva Nair 
DW: openssl-users@openssl.org 
Temat: ODP: CSR creation using pkcs11 dynamic engine

Sorry my bad i was checking bad position in the file...

Od: openssl-users  w imieniu użytkownika 
Piotr Lobacz 
Wysłane: wtorek, 1 czerwca 2021 17:34
Do: Selva Nair 
DW: openssl-users@openssl.org 
Temat: ODP: CSR creation using pkcs11 dynamic engine

Ok not fixed in 0.4.11 it is fixed in master branch... i need to build it from 
master.

Od: openssl-users  w imieniu użytkownika 
Piotr Lobacz 
Wysłane: wtorek, 1 czerwca 2021 17:18
Do: Selva Nair 
DW: openssl-users@openssl.org 
Temat: ODP: CSR creation using pkcs11 dynamic engine

Hi Selva,
btw. i have found one issue in libp11 package 
https://github.com/OpenSC/libp11/issues/304 and i have used 0.4.10 i will check 
firstly with 0.4.11 and give you answer.

BR
Piotr

Od: Selva Nair 
Wysłane: wtorek, 1 czerwca 2021 17:15
Do: Piotr Lobacz 
DW: openssl-users@openssl.org 
Temat: Re: CSR creation using pkcs11 dynamic engine

Hi Piotr,

On Tue, Jun 1, 2021 at 10:57 AM Piotr Lobacz 
mailto:piotr.lob...@softgent.com>> wrote:
Hi,
i have managed to find the engine method static EVP_PKEY *load_privkey(ENGINE 
*engine, const char *s_key_id, UI_METHOD *ui_method, void *callback_data) in 
libp11 package. I have also made a printf callback and i see the output that 
method is being called, but the problem is that i think i need to set this flag 
RSA_FLAG_EXT_PKEY in EVP_PKEY object which i don't know how to do is it even 
possible?

The flag on the key is always set by libp11 (look for RSA_set_key in 
p11_rsa.c). What it doesn't set is any flags on the method -- which you wanted 
to satisfy dotnet. For testing you could add it -- look for 
PKCS11_get_rsa_method in the same file.

Selva
[https://softgent.com/wp-content/uploads/2020/01/Zasob-14.png]<https://www.softgent.com>

Softgent Sp. z o.o., Budowlanych 31d, 80-298 Gdansk, POLAND

KRS: 674406, NIP: 9581679801, REGON: 367090912

www.softgent.com

Sąd Rejonowy Gdańsk-Północ w Gdańsku, VII Wydział Gospodarczy Krajowego 
Rejestru Sądowego

KRS 674406, Kapitał zakładowy: 25 000,00 zł wpłacony w całości.

ODP: CSR creation using pkcs11 dynamic engine

[Piotr Lobacz]

Sorry my bad i was checking bad position in the file...

Od: openssl-users  w imieniu użytkownika 
Piotr Lobacz 
Wysłane: wtorek, 1 czerwca 2021 17:34
Do: Selva Nair 
DW: openssl-users@openssl.org 
Temat: ODP: CSR creation using pkcs11 dynamic engine

Ok not fixed in 0.4.11 it is fixed in master branch... i need to build it from 
master.

Od: openssl-users  w imieniu użytkownika 
Piotr Lobacz 
Wysłane: wtorek, 1 czerwca 2021 17:18
Do: Selva Nair 
DW: openssl-users@openssl.org 
Temat: ODP: CSR creation using pkcs11 dynamic engine

Hi Selva,
btw. i have found one issue in libp11 package 
https://github.com/OpenSC/libp11/issues/304 and i have used 0.4.10 i will check 
firstly with 0.4.11 and give you answer.

BR
Piotr

Od: Selva Nair 
Wysłane: wtorek, 1 czerwca 2021 17:15
Do: Piotr Lobacz 
DW: openssl-users@openssl.org 
Temat: Re: CSR creation using pkcs11 dynamic engine

Hi Piotr,

On Tue, Jun 1, 2021 at 10:57 AM Piotr Lobacz 
mailto:piotr.lob...@softgent.com>> wrote:
Hi,
i have managed to find the engine method static EVP_PKEY *load_privkey(ENGINE 
*engine, const char *s_key_id, UI_METHOD *ui_method, void *callback_data) in 
libp11 package. I have also made a printf callback and i see the output that 
method is being called, but the problem is that i think i need to set this flag 
RSA_FLAG_EXT_PKEY in EVP_PKEY object which i don't know how to do is it even 
possible?

The flag on the key is always set by libp11 (look for RSA_set_key in 
p11_rsa.c). What it doesn't set is any flags on the method -- which you wanted 
to satisfy dotnet. For testing you could add it -- look for 
PKCS11_get_rsa_method in the same file.

Selva
[https://softgent.com/wp-content/uploads/2020/01/Zasob-14.png]<https://www.softgent.com>

Softgent Sp. z o.o., Budowlanych 31d, 80-298 Gdansk, POLAND

KRS: 674406, NIP: 9581679801, REGON: 367090912

www.softgent.com

Sąd Rejonowy Gdańsk-Północ w Gdańsku, VII Wydział Gospodarczy Krajowego 
Rejestru Sądowego

KRS 674406, Kapitał zakładowy: 25 000,00 zł wpłacony w całości.

ODP: CSR creation using pkcs11 dynamic engine

[Piotr Lobacz]

Ok not fixed in 0.4.11 it is fixed in master branch... i need to build it from 
master.

Od: openssl-users  w imieniu użytkownika 
Piotr Lobacz 
Wysłane: wtorek, 1 czerwca 2021 17:18
Do: Selva Nair 
DW: openssl-users@openssl.org 
Temat: ODP: CSR creation using pkcs11 dynamic engine

Hi Selva,
btw. i have found one issue in libp11 package 
https://github.com/OpenSC/libp11/issues/304 and i have used 0.4.10 i will check 
firstly with 0.4.11 and give you answer.

BR
Piotr

Od: Selva Nair 
Wysłane: wtorek, 1 czerwca 2021 17:15
Do: Piotr Lobacz 
DW: openssl-users@openssl.org 
Temat: Re: CSR creation using pkcs11 dynamic engine

Hi Piotr,

On Tue, Jun 1, 2021 at 10:57 AM Piotr Lobacz 
mailto:piotr.lob...@softgent.com>> wrote:
Hi,
i have managed to find the engine method static EVP_PKEY *load_privkey(ENGINE 
*engine, const char *s_key_id, UI_METHOD *ui_method, void *callback_data) in 
libp11 package. I have also made a printf callback and i see the output that 
method is being called, but the problem is that i think i need to set this flag 
RSA_FLAG_EXT_PKEY in EVP_PKEY object which i don't know how to do is it even 
possible?

The flag on the key is always set by libp11 (look for RSA_set_key in 
p11_rsa.c). What it doesn't set is any flags on the method -- which you wanted 
to satisfy dotnet. For testing you could add it -- look for 
PKCS11_get_rsa_method in the same file.

Selva
[https://softgent.com/wp-content/uploads/2020/01/Zasob-14.png]<https://www.softgent.com>

Softgent Sp. z o.o., Budowlanych 31d, 80-298 Gdansk, POLAND

KRS: 674406, NIP: 9581679801, REGON: 367090912

www.softgent.com

Sąd Rejonowy Gdańsk-Północ w Gdańsku, VII Wydział Gospodarczy Krajowego 
Rejestru Sądowego

KRS 674406, Kapitał zakładowy: 25 000,00 zł wpłacony w całości.

ODP: CSR creation using pkcs11 dynamic engine

[Piotr Lobacz]

Hi Selva,
btw. i have found one issue in libp11 package 
https://github.com/OpenSC/libp11/issues/304 and i have used 0.4.10 i will check 
firstly with 0.4.11 and give you answer.

BR
Piotr

Od: Selva Nair 
Wysłane: wtorek, 1 czerwca 2021 17:15
Do: Piotr Lobacz 
DW: openssl-users@openssl.org 
Temat: Re: CSR creation using pkcs11 dynamic engine

Hi Piotr,

On Tue, Jun 1, 2021 at 10:57 AM Piotr Lobacz 
mailto:piotr.lob...@softgent.com>> wrote:
Hi,
i have managed to find the engine method static EVP_PKEY *load_privkey(ENGINE 
*engine, const char *s_key_id, UI_METHOD *ui_method, void *callback_data) in 
libp11 package. I have also made a printf callback and i see the output that 
method is being called, but the problem is that i think i need to set this flag 
RSA_FLAG_EXT_PKEY in EVP_PKEY object which i don't know how to do is it even 
possible?

The flag on the key is always set by libp11 (look for RSA_set_key in 
p11_rsa.c). What it doesn't set is any flags on the method -- which you wanted 
to satisfy dotnet. For testing you could add it -- look for 
PKCS11_get_rsa_method in the same file.

Selva
[https://softgent.com/wp-content/uploads/2020/01/Zasob-14.png]

Softgent Sp. z o.o., Budowlanych 31d, 80-298 Gdansk, POLAND

KRS: 674406, NIP: 9581679801, REGON: 367090912

www.softgent.com

Sąd Rejonowy Gdańsk-Północ w Gdańsku, VII Wydział Gospodarczy Krajowego 
Rejestru Sądowego

KRS 674406, Kapitał zakładowy: 25 000,00 zł wpłacony w całości.

ODP: CSR creation using pkcs11 dynamic engine

[Piotr Lobacz]

Hi,
i have managed to find the engine method static EVP_PKEY *load_privkey(ENGINE 
*engine, const char *s_key_id, UI_METHOD *ui_method, void *callback_data) in 
libp11 package. I have also made a printf callback and i see the output that 
method is being called, but the problem is that i think i need to set this flag 
RSA_FLAG_EXT_PKEY in EVP_PKEY object which i don't know how to do is it even 
possible?

BR
Piotr Łobacz

Od: openssl-users  w imieniu użytkownika 
Piotr Lobacz 
Wysłane: sobota, 29 maja 2021 20:12
Do: Selva Nair 
DW: openssl-users@openssl.org 
Temat: ODP: CSR creation using pkcs11 dynamic engine

Hi, unfortunately that is not that simple :( These methods are not being 
exposed by the dotnet. Porting them would take to much time because of the 
method struct. Recompiling the whole dotnet sdk is also not an option.

You know, i've been reading your mail and keep thinking and for now i see that 
the fastest way is to simply modify libp11 proxy engine in EVP_load_private_key 
method. First i can verify there the EVP_test_flag on the key and second modify 
the engine flags. This way i will be 100% sure that the problem is on dotnet 
side. Because when i was testing this key on the token i was generating 1024 
bit length key and written it's length to the console. Than i have erased it 
completly and generated a new key pair with modified key length to 2048. The 
output result was changed: 1024 -> 2048. So the conclusion was that the key i 
taken correclty.

I will check this on monday and keep you inform. Have a nice weekend.

BR
Piotr

Od: Selva Nair 
Wysłane: sobota, 29 maja 2021 03:34
Do: Piotr Lobacz 
Temat: Re: CSR creation using pkcs11 dynamic engine

Hi,

I will also check these flags of my RSA object using RSA_test_flags and give 
you the answer. In the meantime as you have already told, the experts in here 
can share their knowledge, but i rather suspect that all you said is correct :] 
and the bug is in the dotnet implementation...

You could probably work around it by getting the method from the key using meth 
= RSA_get_method(rsa) and then setting the flag on the method using 
RSA_meth_set_flags(meth, flags). May not be a nice thing to do to a method 
owned by the engine, but should work if those API are exposed via dotnet.

If this is indeed the problem, you could try lobbying two places: dotnet devs 
to add a check for flags in the key, and libp11/pkcs11 engine devs to also set 
the flags on the method. One of them may oblige, depending on their thoughts on 
what is "right".

Selva
[https://softgent.com/wp-content/uploads/2020/01/Zasob-14.png]<https://www.softgent.com>

Softgent Sp. z o.o., Budowlanych 31d, 80-298 Gdansk, POLAND

KRS: 674406, NIP: 9581679801, REGON: 367090912

www.softgent.com

Sąd Rejonowy Gdańsk-Północ w Gdańsku, VII Wydział Gospodarczy Krajowego 
Rejestru Sądowego

KRS 674406, Kapitał zakładowy: 25 000,00 zł wpłacony w całości.

ODP: CSR creation using pkcs11 dynamic engine

[Piotr Lobacz]

Hi, unfortunately that is not that simple :( These methods are not being 
exposed by the dotnet. Porting them would take to much time because of the 
method struct. Recompiling the whole dotnet sdk is also not an option.

You know, i've been reading your mail and keep thinking and for now i see that 
the fastest way is to simply modify libp11 proxy engine in EVP_load_private_key 
method. First i can verify there the EVP_test_flag on the key and second modify 
the engine flags. This way i will be 100% sure that the problem is on dotnet 
side. Because when i was testing this key on the token i was generating 1024 
bit length key and written it's length to the console. Than i have erased it 
completly and generated a new key pair with modified key length to 2048. The 
output result was changed: 1024 -> 2048. So the conclusion was that the key i 
taken correclty.

I will check this on monday and keep you inform. Have a nice weekend.

BR
Piotr

Od: Selva Nair 
Wysłane: sobota, 29 maja 2021 03:34
Do: Piotr Lobacz 
Temat: Re: CSR creation using pkcs11 dynamic engine

Hi,

I will also check these flags of my RSA object using RSA_test_flags and give 
you the answer. In the meantime as you have already told, the experts in here 
can share their knowledge, but i rather suspect that all you said is correct :] 
and the bug is in the dotnet implementation...

You could probably work around it by getting the method from the key using meth 
= RSA_get_method(rsa) and then setting the flag on the method using 
RSA_meth_set_flags(meth, flags). May not be a nice thing to do to a method 
owned by the engine, but should work if those API are exposed via dotnet.

If this is indeed the problem, you could try lobbying two places: dotnet devs 
to add a check for flags in the key, and libp11/pkcs11 engine devs to also set 
the flags on the method. One of them may oblige, depending on their thoughts on 
what is "right".

Selva
[https://softgent.com/wp-content/uploads/2020/01/Zasob-14.png]

Softgent Sp. z o.o., Budowlanych 31d, 80-298 Gdansk, POLAND

KRS: 674406, NIP: 9581679801, REGON: 367090912

www.softgent.com

Sąd Rejonowy Gdańsk-Północ w Gdańsku, VII Wydział Gospodarczy Krajowego 
Rejestru Sądowego

KRS 674406, Kapitał zakładowy: 25 000,00 zł wpłacony w całości.

ODP: CSR creation using pkcs11 dynamic engine

[Piotr Lobacz]

Hi Selva,
i have found the cause of the problem. It is the HasNoPrivateKey function which 
is in dotnet OpenSsl 
./src/Native/Unix/System.Security.Cryptography.Native/pal_rsa.c implementation. 
The crux of the problem is in these lines:

// The method has descibed itself as having the private key external to the 
structure.
// That doesn't mean it's actually present, but we can't tell.
#pragma clang diagnostic push
#pragma clang diagnostic ignored "-Wcast-qual"
if (RSA_meth_get_flags((RSA_METHOD*)meth) & RSA_FLAG_EXT_PKEY)
#pragma clang diagnostic pop
{
return 0;
}

For some reason i suspect that i do not have this RSA_FLAG_EXT_PKEY flag set. 
Btw. when i am calling ENGINE_load_private_key method, the arguments i'm 
passing are the engine and the key. Both ui_method and callback_data are passed 
as NULLs. I talked with guys from dotnet in here 
https://github.com/dotnet/runtime/issues/53345 and i need to check RSA 
flags.But the thing that puzzles me is that shouldn't that flag be inserted by 
the engine during the call of ENGINE_load_private_key?

BR
Piotr

Od: Selva Nair 
Wysłane: piątek, 28 maja 2021 20:47
Do: Piotr Lobacz 
DW: openssl-users@openssl.org 
Temat: Re: CSR creation using pkcs11 dynamic engine

Hi,

On Fri, May 28, 2021 at 1:44 PM Piotr Lobacz  wrote:
>
> Ok, i have found out that dotnet OpenSsl library has it's own code for 
> verification is key private. For this it needs the whole data of private key 
> from which this method:
>
> static int HasNoPrivateKey(RSA* rsa)
>
>  which is in ./src/Native/Unix/System.Security.Cryptography.Native/pal_rsa.c 
> of dotnet verifies occurence of all private parameters. Unfortunately from 
> what i know private keys are not extractable from tokens because of 
> CKA_EXTRACTABLE=false parameter.
>
> Correct me if i'm wrong but from what i know about openssl, when i'm 
> switching to a closed engine the whole cryptography is being made by the 
> engine module. I think that there should be some other method verifing if key 
> is private. Maybe somebody could give me a hint?

Its not verifying, but signing operation that is failing. This sounds
like something wrong in the way you are using the dotnet interface or
possibly a bug in (or limitation of) that implementation itself. Like
its not meant to be used when keys are "external".  I have no idea
having never used C#.

But you are right, when the private key is loaded through the pkcs11
engine the key is external (can stay non-extractable), and the signing
operation gets delegated to the engine. Are you sure that the pkey
returned by the ENGINE_get_private_key() and rsa handle generated from
that pkey are valid? I see no error checks in your code unless dotnet
will automatically trigger exceptions on error.

You may get more relevant help in the dotnet community.


Selva
[https://softgent.com/wp-content/uploads/2020/01/Zasob-14.png]

Softgent Sp. z o.o., Budowlanych 31d, 80-298 Gdansk, POLAND

KRS: 674406, NIP: 9581679801, REGON: 367090912

www.softgent.com

Sąd Rejonowy Gdańsk-Północ w Gdańsku, VII Wydział Gospodarczy Krajowego 
Rejestru Sądowego

KRS 674406, Kapitał zakładowy: 25 000,00 zł wpłacony w całości.

ODP: CSR creation using pkcs11 dynamic engine

[Piotr Lobacz]

Ok, i have found out that dotnet OpenSsl library has it's own code for 
verification is key private. For this it needs the whole data of private key 
from which this method:

static int HasNoPrivateKey(RSA* rsa)

 which is in ./src/Native/Unix/System.Security.Cryptography.Native/pal_rsa.c of 
dotnet verifies occurence of all private parameters. Unfortunately from what i 
know private keys are not extractable from tokens because of 
CKA_EXTRACTABLE=false parameter.

Correct me if i'm wrong but from what i know about openssl, when i'm switching 
to a closed engine the whole cryptography is being made by the engine module. I 
think that there should be some other method verifing if key is private. Maybe 
somebody could give me a hint?

BR
Piotr

Od: openssl-users  w imieniu użytkownika 
Piotr Lobacz 
Wysłane: piątek, 28 maja 2021 13:10
Do: openssl-users@openssl.org 
Temat: CSR creation using pkcs11 dynamic engine

Hi all,
i'm trying to generate CSR using C# System.SecurityCryptography.Openssl library 
together with pkcs11 token library. The whole proces for this in command line 
works without any problems. For execution of this process i use command:

openssl req -new -subj '/C=PL/ST=Gdansk/L=Gdansk/CN=softgent.com/' -sha256 
-engine pkcs11 -keyform engine -key 
"pkcs11:token=foo;object=tls;type=private;pin-value=1234567890"

The CSR is being generated and the output is like this:

-BEGIN CERTIFICATE REQUEST-
MIIBADCBqAIBADBGMQswCQYDVQQGEwJQTDEPMA0GA1UECAwGR2RhbnNrMQ8wDQYD
VQQHDAZHZGFuc2sxFTATBgNVBAMMDHNvZnRnZW50LmNvbTBZMBMGByqGSM49AgEG
CCqGSM49AwEHA0IABB7SwUzg8S+3iYNiqGPlidqwCdmuY8MV3RfKDiR5tL/I//Cn
9dGCBAfxTO23gb5pygIXB/qCARYuYLiGpE+tFo+gADAKBggqhkjOPQQDAgNHADBE
AiAI4kDGjeO/V3f7RWe34e00aZAubjLGuIRbxgmQosu7mQIgQDK3Nx22fJn80Cml
t3EQTa6x9oC4RtibFgWCxZ36Wyo=
-END CERTIFICATE REQUEST-

Now i'm trying to do all that programatically. In order to do that i have added 
some OpenSsl C# missing support for the engines and used the 
ENGINE_load_private_key method to retrieve SafeEvpPKeyHandle which is being 
retrieved (i have checked it with changing the key id value). The key which i'm 
using is "label_" + myKeyId i.e. "label_tls". The code looks like this:

public virtual SafeEvpPKeyHandle GetPrivKey(string label)
{
string keyId = "label_" + label;
SafeEvpPKeyHandle pkey = SafeNativeMethods.ENGINE_load_private_key(engine, 
keyId, IntPtr.Zero, IntPtr.Zero);
if(pkey.IsInvalid)
{
 throw new InvalidOperationException("engine: unable to find private 
key with label='{label}'");
}

return pkey;
}

This is being returnin me SafeEvpPKeyHandle. The problem is in calling 
CreateSigningRequest from System.Security.Cryptography.OpenSsl.dll. I have this 
method:

public virtual string GetCSR(SafeEvpPKeyHandle pkey, string ext, 
HashAlgorithmName name)
{
// FIXME: determine key type
RSA rsa = new RSAOpenSsl(pkey);

CertificateRequest req = new CertificateRequest("CN=potato", rsa, name, 
RSASignaturePadding.Pkcs1); // this method is only for RSA key different is for 
EC, DSA etc.
byte[] requestDer = req.CreateSigningRequest();
string requestPem = new string(PemEncoding.Write("CERTIFICATE REQUEST", 
requestDer));
return requestPem;
}

and i'm getting this error:

Unhandled exception. Interop+Crypto+OpenSslCryptographicException: 
error:04075093:rsa routines:RSA_sign:value missing
   at System.Security.Cryptography.RSAOpenSsl.TrySignHash(ReadOnlySpan`1 hash, 
Span`1 destination, HashAlgorithmName hashAlgorithm, RSASignaturePadding 
padding, Boolean allocateSignature, Int32& bytesWritten, Byte[]& signature)
   at System.Security.Cryptography.RSAOpenSsl.SignHash(Byte[] hash, 
HashAlgorithmName hashAlgorithm, RSASignaturePadding padding)
   at System.Security.Cryptography.RSA.SignData(Byte[] data, Int32 offset, 
Int32 count, HashAlgorithmName hashAlgorithm, RSASignaturePadding padding)
   at System.Security.Cryptography.RSA.SignData(Byte[] data, HashAlgorithmName 
hashAlgorithm, RSASignaturePadding padding)
   at 
System.Security.Cryptography.X509Certificates.RSAPkcs1X509SignatureGenerator.SignData(Byte[]
 data, HashAlgorithmName hashAlgorithm)
   at 
System.Security.Cryptography.X509Certificates.Pkcs10CertificationRequestInfo.ToPkcs10Request(X509SignatureGenerator
 signatureGenerator, HashAlgorithmName hashAlgorithm)
   at 
System.Security.Cryptography.X509Certificates.CertificateRequest.CreateSigningRequest(X509SignatureGenerator
 signatureGenerator)
   at 
System.Security.Cryptography.X509Certificates.CertificateRequest.CreateSigningRequest()
   at System.Security.Cryptography.Engine.GetCSR(SafeEvpPKeyHandle pkey, String 
ext, HashAlgorithmName name) in 
/home/plobacz/workspace/OpenSsl.DynamicEngine/Engine.cs:line 72
   at Flexgent.Services.CryptoSubsystem.CryptoSubsystem.Configure(String 
config) in 
/home/plobacz/workspace/crypto-subsystem/flexgent/extensions/security/crypto-subsystem/src/CryptoSubsystem.cs:line
 145
   at