Hi,
Our application experienced a serious performance drop in FIPS mode. The
connection per second (CPS) dropped about 50% in FIPS mode than in non-FIPS
mode.
We run the oprofile and find that there are lock contentions in FIPS mode:
FIPS oprofile system wide report
samples % app name symbol name
6930732 77.6544 no-vmlinux /no-vmlinux
269911 3.0242 libpthread-2.12.so pthread_rwlock_unlock <----
233107 2.6118 libpthread-2.12.so pthread_rwlock_rdlock <----
129953 1.4560 libpthread-2.12.so __lll_lock_wait
113393 1.2705 libpython2.6.so.1.0 /usr/lib64/libpython2.6.so.1.0
85393 0.9568 sport bn_sqr4x_mont
64619 0.7240 sport locking_function(int, int, char
const*, int) <--- our application
Running oprofile on our application shows that the locking_function is
intensively called by FIPS_selftest_failed/FIPS_module_mode:
-------------------------------------------------------------------------------
…
277 8.1018 CRYPTO_add_lock
1406 41.1231 FIPS_selftest_failed <-----
1555 45.4811 FIPS_module_mode <-----
3633 4.5420 locking_function(int, int, char const*, int)
3633 100.000 locking_function(int, int, char const*, int) [self]
-------------------------------------------------------------------------------
Read the source code and find FIPS_selftest_failed/FIPS_module_mode call
pthread_rwlock_rdlock/pthread_rwlock_unlock on two FIPS mutex:
CRYPTO_LOCK_FIPS2/CRYPTO_LOCK_FIPS
After put the following test code in the locking_function, the CPS raised up
around 40~50%, close to the CPS in non-FIPS mode:
static void
locking_function(int mode, int n, const char * file, int line)
{
if( n == CRYPTO_LOCK_FIPS2 || n == CRYPTO_LOCK_FIPS ) {
return;
}
…
}
But we cannot find a good explanation for this. Seems
CRYPTO_LOCK_FIPS2/CRYPTO_LOCK_FIPS are only pthread_rwlock_wrlock/unlock in the
FIPS initialization stage. How come the threads can get blocked on
pthread_rwlock_rdlock/unlock on the two mutexes after FIPS already be
initialized? Does anyone know if the two mutexes get
pthread_rwlock_rdlock/unlock on other place after initialization stage? Or help
point out what may be the root cause for this issue?
Thanks,
--Chang
