What do you refer when you say to expand the 40 bits correctly?, because the unwrapped key has 5 bytes, and the effective bits of the RC2 key are 40. Other times we could have a 80 bits key and only 40 of them were effectives, but now?.
Regards, Antonio.
You question indicates that key expansion is almost certainly your problem.
RC2 takes the key, loads it into a larger array (128 byte I believe), then uses the key to derive the remainder of the bytes in the array.
It then masks off all but the last N bytes, and again refills mask portion of the array using the given N bytes. The array is then used as
the 'key' in the RC2 algorithm. N is the same as 'effective' key size. This is because even though the user needs the full 80 bits to correctly decrypt the message, and attacker only needs to find the 40 bit value which occurs before the 'masking' operation. He can decrypt the message cycling through the 1024 bit key space from key 0000.... 00000 to key 0000.... fffff (a 40 bit key space).
If your 16 byte key is working, it probably means your implimentation has 'N' set to 128 (fixed) rather than depended on the effective key size.
bob
smime.p7s
Description: S/MIME Cryptographic Signature
