> padding = 3 means "no padding" indicating that the data for signature is > already padded. That's why the data size (flen) is 256 (hashed data padded to > the rsa key size of 2048 bits, I guess). If you are using OpenSSL 1.1.1, this > could be due to PSS padding in which case current implementation passes > pre-padded data for raw signature to the callback. AFAIK, pkcs11-helper only > handles PKCS1 padding (CKM_RSA_PKCS) though pkcs11 standard does support raw > signatures.
https://github.com/OpenSC/pkcs11-helper/blob/0e2ae10ef9611beef92457171e8c78d8e936dfca/lib/pkcs11h-openssl.c#L570 if (padding != RSA_PKCS1_PADDING) { rv = CKR_MECHANISM_INVALID; goto cleanup; }