The issue is that checking RSA_FLAG_CACHE_PRIVATE was a hack. This flag doesn't really say that this is a private key but rather that caching should be used in private key operations. It worked in your case and didn't work for someone else.
As I suggested in another reply, I think you should mark the key as private yourself. This is a better way to do it. Aleksey On 9/17/17 10:36 AM, Pablo Gabriel Gallardo wrote: > 2017-09-17 13:46 GMT-03:00 Pablo Gabriel Gallardo <[email protected]>: >> I'll investigate to check what else can we do to determine whether an >> EVP_PKEY is private or not. I'm not an OpenSSL expert but I want to help >> with that. >> >> Regards, >> >> Pablo G. Gallardo > > Aleksey, > > I have a question. This is the code: > > ```c > RSA_get0_key(rsa, &n, &e, &d); > if(n != NULL && e != NULL) { > if(d != NULL) { > return(xmlSecKeyDataTypePrivate | xmlSecKeyDataTypePublic); > } else if(RSA_test_flags(rsa, (RSA_FLAG_EXT_PKEY | > RSA_FLAG_CACHE_PRIVATE)) != 0) { > /* > * !!! HACK !!! Also see DSA key > * We assume here that engine *always* has private key. > * This might be incorrect but it seems that there is no > * way to ask engine if given key is private or not. > */ > return(xmlSecKeyDataTypePrivate | xmlSecKeyDataTypePublic); > } else { > return(xmlSecKeyDataTypePublic); > } > } > ``` > > First we check whether d is NULL or not `if(d != NULL)`. If we are > dealing with a public key generally d is, indeed, NULL. In the case of > smartkeys, even if we are dealing with a private key d is also NULL > because d is inside the smartkey (never transmitted to the memory or > CPU). > > So we are failing in the second condition `RSA_test_flags(rsa, > (RSA_FLAG_EXT_PKEY | RSA_FLAG_CACHE_PRIVATE)) != 0`, the question is: > Those users that are reporting problems, What type of key they are > using? If they are using a private key in a file, how can d be NULL? > And if they are using a private key in another device, how they were > doing that before the change in the condition so I can do the same? > > Best, > > Pablo G. Gallardo > _______________________________________________ xmlsec mailing list [email protected] http://www.aleksey.com/mailman/listinfo/xmlsec
