I am trying to build a DLL that includes the OpenSSL FIPS Object Module, and then load that DLL from my application. This is on Windows 7 64-bit using Microsoft Visual Studio v10. The DLL is built successfully and my application can load it, but when I call the FIPS_mode_set(1) function, the self-test fails.
Here is how I'm building everything: > cd openssl-fips-2.0.5 > C:\Program Files (x86)\Microsoft Visual Studio 10.0\vc\bin\amd64\vcvars64.bat > ms\do_fips.bat > cd ..\openssl-1.0.1e > perl Configure VC-WIN64A fips --with-fipslibdir ..\openssl-fips-2.0.5 > ms\do_win64a.bat > nmake -f ms\nt.mak Then when building my DLL, I'm including the ssleay32.lib, libeayfips32.lib, and libeaycompat32.lib files from the openssl-1.0.1e\out32 directory and running the following batch file instead of link.exe: > call C:\Program Files (x86)\Microsoft Visual Studio > 10.0\vc\bin\amd64\vcvars64.bat > set FIPS_LINK=link > set FIPS_CC=cl > set FIPS_CC_ARGS=/Foobj\fips_premain.obj -MD -c > set PREMAIN_DSO_EXE=openssl-1.0.1e\out32\fips_premain_dso.exe > set FIPS_TARGET=mydll.dll > set FIPS_SHA1_EXE=openssl-fips-2.0.5\out32dll\fips_standalone_sha1.exe > set FIPSLIB_D=openssl-fips-2.0.5\out32dll > perl openssl-fips-2.0.5\util\fipslink.pl /map /out:mydll.dll > obj\fips_premain.obj @mydll.lnk Where mydll.lnk contains all the standard link directives we usually use. This succeeds, and I end up with a DLL that is functional. In fact, if I don't call FIPS_mode_set (1), I can make TLS connections to remote servers and such. But FIPS_mode_set(1) always fails. When I debug into it, the FIPS_selftest() routine fails because in FIPS_check_incore_fingerprint, the FIPS_incore_fingerprint call returns a different signature than the one used during the build procedure. What's the best way to determine why this is failing? Is there something missing from my build procedure, or some other gotcha I should be looking for? Thanks for any help. Graeme Perrow
