For legacy and compatibility reasons I have to use an older version of the Windows DDK to compile certain libraries, including OpenSSL. I am compiling it with version 3790.1830. My main project (the one that links against OpenSSL) is compiled in Visual Studio 2010.
I spent the morning tracing a crash in BN_nist_mod_384 which I believe is
caused by an optimisation by the DDK compiler, but is easily fixable.
If looking at the crypto/bn/bn_nist.c file distributed with version 1.0.1c
of the library, on lines 1029-1030:
res = (BN_ULONG *)(((PTR_SIZE_INT)c_d&~mask) |
((PTR_SIZE_INT)r_d&mask));
Because c_d is a stack array and not an actual pointer, the DDK compiler is
retrieving the value of the first item in the stack array (in my case it's
0x01) and AND'ing it with (~mask), which is 0xffffffff. This results in a
crash later on when *(0x00000001) is dereferenced.
In assembly it looks like:
MOV eax, DWORD PTR [mask]
NOT eax
AND eax, DWORD PTR [c_d]
At this point [eax] contains 0x01.
Note that this only happens in Debug on Win32; it does not happen in
Release, and does not happen in x64 at all. I haven't spent much time to
investigate why that is.
The fix (in my case) is very simple:
// top of function:
// Name a temporary variable to hold the pointer to c_d
PVOID tmpC;
// new line 1029:
tmpC = (PVOID)((PTR_SIZE_INT) c_d);
res = (BN_ULONG *)(((PTR_SIZE_INT)tmpC&~mask) |
((PTR_SIZE_INT)r_d&mask));
This disables the DDK's "optimisation by loading the effective address (LEA
instruction) of "c_d" into an interim register, and performing the AND
operation on that.
I assume the same applies to the other key sizes but I haven't checked.
Cheers
For legacy and compatibility reasons I have to use an older version of the Windows DDK to compile certain libraries, including OpenSSL.? I am compiling it with version 3790.1830.? My main project (the one that links against OpenSSL) is compiled in Visual Studio 2010.I spent the morning tracing a crash in BN_nist_mod_384 which I believe is caused by an optimisation by the DDK compiler, but is easily fixable.
If looking at the crypto/bn/bn_nist.c file distributed with version 1.0.1c of the library, on lines 1029-1030:
res = (BN_ULONG *)(((PTR_SIZE_INT)c_d&~mask) |
??? ((PTR_SIZE_INT)r_d&mask));
Because c_d is a stack array and not an actual pointer, the DDK compiler is retrieving the value of the first item in the stack array (in my case it's 0x01) and AND'ing it with (~mask), which is 0xffffffff.? This results in a crash later on when *(0x00000001) is dereferenced.
In assembly it looks like:
MOV eax, DWORD PTR [mask]
NOT eax
AND eax, DWORD PTR [c_d]
At this point [eax] contains 0x01.
Note that this only happens in Debug on Win32; it does not happen in Release, and does not happen in x64 at all.? I haven't spent much time to investigate why that is.
The fix (in my case) is very simple:
// top of function:
// Name a temporary variable to hold the pointer to c_d
PVOID tmpC;
// new line 1029:
tmpC = (PVOID)((PTR_SIZE_INT) c_d);
res = (BN_ULONG *)(((PTR_SIZE_INT)tmpC&~mask) |
??? ((PTR_SIZE_INT)r_d&mask));
This disables the DDK's "optimisation by loading the effective address (LEA instruction) of "c_d" into an interim register, and performing the AND operation on that.
I assume the same applies to the other key sizes but I haven't checked.
Cheers
