[openssl-users] FIPS: SSL 3.0 now forbidden in latest NDCPP update
Hi, ... Along with TLS 1.0 (which is absent from OpenSSL FIPS mode) https://www.niap-ccevs.org/pp/pp.cfm?id=CPP_ND_V1.0 Specifically: FCS_TLSS_EXT.1.2 The TSF shall deny connections from clients requesting SSL 1.0, SSL 2.0, SSL 3.0, TLS 1.0 FCS_TLSS_EXT.2.2 The TSF shall deny connections from clients requesting SSL 1.0, SSL 2.0, SSL 3.0, TLS 1.0 In this case, would it be possible to simply compile OpenSSL without support for SSL 3.0, while having FIPS mode taking care of the rest ? I do not remeber the exact option now, although I'm almost sure there's a compile option to exclude SSL 3.0. Am I right and would that work ? Regards. -- View this message in context: http://openssl.6102.n7.nabble.com/FIPS-SSL-3-0-now-forbidden-in-latest-NDCPP-update-tp57695.html Sent from the OpenSSL - User mailing list archive at Nabble.com. ___ openssl-users mailing list To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-users
[openssl-users] ECDSA digest configurations
Hello openssl-users, I have an issue with update of ECDSA digests in our environment. We have our own digest functions for init, update and final where we registered with these functions for NID_sha1, NID_sha256, NID_sha384 and NID_sha512. These digests were updated at openssl, via ENGINE_set_digests() function. After update, i see that all RSA cerificate based ciphers are working fine with both TLS1.0 and TLS1.2. And, i added the ECDSA algorithm with openssl function EVP_add_digest(EVP_ecdsa()); But, for me ECDSA cert based cipher suites are not working. These are landing to following errors. digital envelope routines:EVP_SignFinal:wrong public key type : crypto/evp/p_sign .c:139: SSL routines:SSL3_SEND_SERVER_KEY_EXCHANGE:EVP lib: ssl/s3_srvr.c:2012 Then i tried following Tried to set only NID_ecdsa_with_SHA1 with our init, update and final function and tried to update to openssl via ENGINE_set_digests(). But, now seems to be since i updated NID_ecdsa_with_SHA1 with our sha1 functions, whole TLS1.0 handshakes stopped working with decrypt error or bac mac record errors. Based on this, seems to be there is a basic thing am missing while configuring ECDSA based digests. Can you please help me on this configuration? How can we differentiate between ECDSA digests to RSA digests? Because both usually lands on to same SHA1, SHA256 etc functions. I tried one more thing that, along with NID_sha1, NID_sha256, NID_sha384, NID_sha512 i tried to define NID_ecdsa_with_SHA1, NID_ecdsa_with_SHA256, NID_ecdsa_with_SHA384 and NID_ecdsa_with_SHA512 with respective settings. But, with this setting also landing to same above error of wrong public key type. I need your inputs on this issue. Thanks, Rajeswari. ___ openssl-users mailing list To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-users
[openssl-users] Web Services Security: SOAP Message Security 1.0 (WS-Security 2004)” Signature Value
What OpenSSL command is required to generate a proper “Web Services Security: SOAP Message Security 1.0 (WS-Security 2004)” Signature Value I have the following guidelines in generating the SignatureValue BinarySecurityToken u:Id=* ValueType= http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-x509-token-profile-1.0#X509v3 EncodingType= http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-soap-message-security-1.0#Base64Binary CERTIFICATE/o:BinarySecurityToken I have a self signed .p12 and/or .pem Certificate file to use in the encryption. I have the string that needs to be encrypted as the SignatureValue. I suspect that since the Value type is X509v3 that this is a clue on how to generate the Signature. I am trying to build a Web Service client within a C++ application which uses OpenSSL for encryption and gSOAP to implement the SOAP protocol. Any help would be appreciated. Thanks John Gerard ___ openssl-users mailing list To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-users
Re: [openssl-users] Performance problems with OpenSSL and threading
Some of the algorithms still invoke the FIPS flavor of the algorithm
even when FIPS is disabled. For example, this code is from
EVP_DigestUpdate().
int EVP_DigestUpdate(EVP_MD_CTX *ctx, const void *data, size_t count)
{
#ifdef OPENSSL_FIPS
return FIPS_digestupdate(ctx, data, count);
#else
return ctx-update(ctx, data, count);
#endif
}
In 1.0.1, locks 39 and 40 are:
# define CRYPTO_LOCK_FIPS39
# define CRYPTO_LOCK_FIPS2 40
You should try adding some logic to skip the lock in your lock handler
when the lock ID is 39 or 40. Again, it should be safe to not lock on
these two as long as the FIPS POST has completed.
On 04/24/2015 05:56 PM, Bryan Call wrote:
In my last email I ran the benchmark on Fedora 21 (big mistake).
Here are the results when running it back on the 28/56 core RHEL 6.5
server showing contention in a different place (fips). Is there a
reason it would be calling into the FIPS code to get a lock of FIPS is
not enabled?
Contention log:
[Apr 24 21:35:09.731] Server {0x7f5529ccd700} ERROR: contention for
lock - total contention: 70662 waiting: 16 file: fips.c line: 471 type: 40
[Apr 24 21:35:09.732] Server {0x7f55299ca700} ERROR: contention for
lock - total contention: 71605 waiting: 2 file: fips.c line: 107 type: 39
Line in in fips that see the contention the most:
[bcall@bm77 trafficserver]$ grep fips.c diags.log | cut -f 19 -d ' ' |
sort | uniq -c | sort -rn
875379 471
288338 107
275472 127
Different lock types and their contention frequency (multiple benchmarks):
875379 40 - in fips.c
563810 39 - also in fips.c
440518 1
44145 18
17447 2
11356 10
1377 19
687 12
532 25
302 33
239 9
58 16
40 21
32 11
19 3
1 14
Code to diable FIPS and log:
#ifdef OPENSSL_FIPS
int mode = FIPS_mode();
Debug(ssl, FIPS_mode: %d, mode);
mode = FIPS_mode();
if (mode) {
FIPS_mode_set(0);
Debug(ssl, FIPS_mode: %d, mode);
}
#endif
[Apr 24 21:43:45.860] Server {0x7f7628146800} DEBUG: (ssl) FIPS_mode: 0
-Bryan
On Apr 24, 2015, at 10:56 AM, John Foley fol...@cisco.com
mailto:fol...@cisco.com wrote:
When you create the private key given to SSL_CTX, be sure to
instantiate a private key for each thread as well. If you share the
same key across all threads, this can introduce contention.
Here are the type 1 2 locks:
# define CRYPTO_LOCK_ERR 1
# define CRYPTO_LOCK_EX_DATA 2
Have you checked the OpenSSL error log to see what errors are being
generated?
Also, if you disable FIPS, does the thread contention still occur?
On 04/24/2015 01:36 PM, Bryan Call wrote:
We are using a single SSL_CTX across all the threads, so I will
create multiple SSL_CTX per thread. I implemented dynlock callbacks
yesterday, but I didn’t see them being used in the tests I did. I
also added atomic counters to see what type of locks are having
contention (see logs below):
Most of the lock contention seems to be around type 2 and type 1
locks when calling our lock callback. In the FIPS/POST and SSL_CTX
recommendations, in your previous email, what types of locks would
those help with?
Log output:
[Apr 24 10:21:24.339] Server {0xb31} ERROR: SSLUtils.cc
http://sslutils.cc/:178 (SSL_locking_callback) contention for
lock - total contention: 3938 waiting: 1 file:
/SourceCache/OpenSSL098/OpenSSL098-52.20.2/src/crypto/ex_data.c
line: 496 type: 2
[Apr 24 10:21:24.754] Server {0xb10a000} ERROR: SSLUtils.cc
http://sslutils.cc/:178 (SSL_locking_callback) contention for
lock - total contention: 1350 waiting: 1 file:
/SourceCache/OpenSSL098/OpenSSL098-52.20.2/src/crypto/err/err_def.c
line: 343 type: 1
-Bryan
On Apr 23, 2015, at 4:46 PM, John Foley fol...@cisco.com
mailto:fol...@cisco.com wrote:
Looking at your call stack, it appears you're testing with FIPS
mode enabled. There's a lock in FIPS_selftest_failed() that's used
to ensure the crypto algorithms inside the FIPS boundary are not
utilized until after the POST has completed and passed.
Unfortunately this lock remains installed for the lifetime of the
process. We've seen thread contention in this area when using
OpenSSL. One work-around is to modify your handler that you pass
to SSL_locking_callback() to not lock when the lock ID is
CRYPTO_LOCK_FIPS. But you would only want to disable this
particular lock ID after the POST has completed.
If the above doesn't help, another item to check is whether you're
instantiating an SSL_CTX for each thread. I've seen some
developers use a single SSL_CTX for multiple threads. This results
in a single instance of the server cert and private key being
shared across all the threads. This results in thread contention
on the asymmetric key operations associated with the cert and
private key (e.g. RSA). Be sure that each thread instantiates it's
own SSL_CTX and certificate/private
[openssl-users] Can RSA_private_decrypt succeed with the wrong padding?
Using OpenSSL 1.0.1m on 64-bit Windows and Linux. I have implemented RSA encryption using the RSA_public_encrypt and RSA_private_decrypt functions and various padding types. This is working fine except that in very rare cases, my test fails because decrypting succeeds when it should fail. I'm basically doing this (pseudocode): RSA_public_encrypt( abc, encrypted_data, my_public_key, RSA_PKCS1_OAEP_PADDING ); RSA_private_decrypt( encrypted_data, decrypted_data, my_private_key, RSA_NO_PADDING ); Note that the padding types are different. The vast majority of the time, I get an error from the RSA_private_decrypt call but now and again, it succeeds. The output data is NOT abc however, it's seemingly random data. I've seen varying lengths for the output data as well, from 11 bytes to 115. I don't understand the underlying crypto well enough to know - is it possible for RSA_private_decrypt to be unable to tell that the wrong padding was used, or is this a bug in OpenSSL? Thanks Graeme Perrow ___ openssl-users mailing list To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-users
[openssl-users] How do I uninitialize OpenSSL properly?
Hello everyone, I am using OpenSSL sockets and I have the problem I described in the following thread, i.e. a bunch of memory-leaks that happen most likely because I don't uninitialize OpenSSL properly. I believe this to be the cause, because the errors appeared exactly at the moment when I switched from dynamic to static linking of libeay32 and ssleay32: http://stackoverflow.com/questions/29845527/how-to-properly-uninitialize-openssl I admit I am still a beginner, but please help me anyway. :-) I swear I have been googling and fumbling around with OpenSSL quite a bit by myself so far. Cheers Alex ___ openssl-users mailing list To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-users
Re: [openssl-users] Web Services Security: SOAP Message Security 1.0 (WS-Security 2004)? Signature Value
On Fri, Apr 24, 2015 at 03:55:18PM +, Salz, Rich wrote: To generate the signature look at the pkeyutl app. (Sorry, don?t remember any other details) Typically, signatures are created with: openssl dgst -sha256 -sign key.pem ... See the dgst(1) manpage. -- Viktor. ___ openssl-users mailing list To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-users
Re: [openssl-users] Web Services Security: SOAP Message Security 1.0 (WS-Security 2004)” Signature Value
WS-Security? Blast from the past … ☺ To fill in the BinarySecurityToken field, take the PEM version of the cert, strip off the leading and trailing marker lines ---BEGIN… and output that. To generate the signature look at the pkeyutl app. (Sorry, don’t remember any other details) -- Senior Architect, Akamai Technologies IM: richs...@jabber.at Twitter: RichSalz ___ openssl-users mailing list To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-users
Re: [openssl-users] Can RSA_private_decrypt succeed with the wrong padding?
On Sat, Apr 25, 2015 at 12:49:21AM +, Perrow, Graeme wrote: Using OpenSSL 1.0.1m on 64-bit Windows and Linux. I have implemented RSA encryption using the RSA_public_encrypt and RSA_private_decrypt functions and various padding types. This is working fine except that in very rare cases, my test fails because decrypting succeeds when it should fail. I'm basically doing this (pseudocode): RSA_public_encrypt( abc, encrypted_data, my_public_key, RSA_PKCS1_OAEP_PADDING ); RSA_private_decrypt( encrypted_data, decrypted_data, my_private_key, RSA_NO_PADDING ); A real code fragment would be substantially more useful that pseudo-code here. This should *always* succeed, provided you pass the correct length to RSA_private_decrypt. From the manpage: flen must be ... and exactly RSA_size(rsa) for RSA_NO_PADDING. Note that the padding types are different. The vast majority of the time, I get an error from the RSA_private_decrypt call but now and again, it succeeds. You're doing something wrong, it should always recover the OAEP padded data, which is basically random, you need to reverse the OAEP padding to recover the plaintext. I don't understand the underlying crypto well enough to know - is it possible for RSA_private_decrypt to be unable to tell that the wrong padding was used, or is this a bug in OpenSSL? When not using padding, that's not wrong padding, you're just doing a raw RSA decrypt, and any input that is smaller than the modulus (but has the same bit length) should decrypt to something. -- Viktor. ___ openssl-users mailing list To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-users
Re: [openssl-users] FIPS: SSL 3.0 now forbidden in latest NDCPP update
On Fri, Apr 24, 2015, jonetsu wrote: ... Along with TLS 1.0 (which is absent from OpenSSL FIPS mode) https://www.niap-ccevs.org/pp/pp.cfm?id=CPP_ND_V1.0 Specifically: FCS_TLSS_EXT.1.2 The TSF shall deny connections from clients requesting SSL 1.0, SSL 2.0, SSL 3.0, TLS 1.0 FCS_TLSS_EXT.2.2 The TSF shall deny connections from clients requesting SSL 1.0, SSL 2.0, SSL 3.0, TLS 1.0 In this case, would it be possible to simply compile OpenSSL without support for SSL 3.0, while having FIPS mode taking care of the rest ? I do not remeber the exact option now, although I'm almost sure there's a compile option to exclude SSL 3.0. Am I right and would that work ? In FIPS mode SSL 3.0 is not allowed: that has always been the case. TLS 1.0 is currently permitted though. Steve. -- Dr Stephen N. Henson. OpenSSL project core developer. Commercial tech support now available see: http://www.openssl.org ___ openssl-users mailing list To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-users
Re: [openssl-users] ECDSA digest configurations
On Fri, Apr 24, 2015, Rajeswari K wrote: Hello openssl-users, I have an issue with update of ECDSA digests in our environment. We have our own digest functions for init, update and final where we registered with these functions for NID_sha1, NID_sha256, NID_sha384 and NID_sha512. These digests were updated at openssl, via ENGINE_set_digests() function. After update, i see that all RSA cerificate based ciphers are working fine with both TLS1.0 and TLS1.2. And, i added the ECDSA algorithm with openssl function EVP_add_digest(EVP_ecdsa()); But, for me ECDSA cert based cipher suites are not working. These are landing to following errors. digital envelope routines:EVP_SignFinal:wrong public key type : crypto/evp/p_sign .c:139: SSL routines:SSL3_SEND_SERVER_KEY_EXCHANGE:EVP lib: ssl/s3_srvr.c:2012 Then i tried following Tried to set only NID_ecdsa_with_SHA1 with our init, update and final function and tried to update to openssl via ENGINE_set_digests(). But, now seems to be since i updated NID_ecdsa_with_SHA1 with our sha1 functions, whole TLS1.0 handshakes stopped working with decrypt error or bac mac record errors. Based on this, seems to be there is a basic thing am missing while configuring ECDSA based digests. Can you please help me on this configuration? How can we differentiate between ECDSA digests to RSA digests? Because both usually lands on to same SHA1, SHA256 etc functions. I tried one more thing that, along with NID_sha1, NID_sha256, NID_sha384, NID_sha512 i tried to define NID_ecdsa_with_SHA1, NID_ecdsa_with_SHA256, NID_ecdsa_with_SHA384 and NID_ecdsa_with_SHA512 with respective settings. But, with this setting also landing to same above error of wrong public key type. I need your inputs on this issue. So does your code just perform the digest operation and not signing? Which version was it working with before? There isn't really any difference between an ECDSA and RSA digest. Older versions of OpenSSL linked digests and signing algorithms. That link no longer exists and some algorithms are marked as being suitable for multiple public key algorithms. Are you setting the flag EVP_MD_FLAG_PKEY_METHOD_SIGNATURE in the EVP_MD structure? An ENGINE providing alternative implementations of built in digests shoudn't need to call EVP_add_digest. If everything is set up properly OpenSSL should just switch to the ENGINE implementation. Steve. -- Dr Stephen N. Henson. OpenSSL project core developer. Commercial tech support now available see: http://www.openssl.org ___ openssl-users mailing list To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-users
Re: [openssl-users] Performance problems with OpenSSL and threading
We are using a single SSL_CTX across all the threads, so I will create multiple
SSL_CTX per thread. I implemented dynlock callbacks yesterday, but I didn’t
see them being used in the tests I did. I also added atomic counters to see
what type of locks are having contention (see logs below):
Most of the lock contention seems to be around type 2 and type 1 locks when
calling our lock callback. In the FIPS/POST and SSL_CTX recommendations, in
your previous email, what types of locks would those help with?
Log output:
[Apr 24 10:21:24.339] Server {0xb31} ERROR: SSLUtils.cc
http://sslutils.cc/:178 (SSL_locking_callback) contention for lock - total
contention: 3938 waiting: 1 file:
/SourceCache/OpenSSL098/OpenSSL098-52.20.2/src/crypto/ex_data.c line: 496 type:
2
[Apr 24 10:21:24.754] Server {0xb10a000} ERROR: SSLUtils.cc
http://sslutils.cc/:178 (SSL_locking_callback) contention for lock - total
contention: 1350 waiting: 1 file:
/SourceCache/OpenSSL098/OpenSSL098-52.20.2/src/crypto/err/err_def.c line: 343
type: 1
-Bryan
On Apr 23, 2015, at 4:46 PM, John Foley fol...@cisco.com
mailto:fol...@cisco.com wrote:
Looking at your call stack, it appears you're testing with FIPS mode enabled.
There's a lock in FIPS_selftest_failed() that's used to ensure the crypto
algorithms inside the FIPS boundary are not utilized until after the POST has
completed and passed. Unfortunately this lock remains installed for the
lifetime of the process. We've seen thread contention in this area when
using OpenSSL. One work-around is to modify your handler that you pass to
SSL_locking_callback() to not lock when the lock ID is CRYPTO_LOCK_FIPS. But
you would only want to disable this particular lock ID after the POST has
completed.
If the above doesn't help, another item to check is whether you're
instantiating an SSL_CTX for each thread. I've seen some developers use a
single SSL_CTX for multiple threads. This results in a single instance of
the server cert and private key being shared across all the threads. This
results in thread contention on the asymmetric key operations associated with
the cert and private key (e.g. RSA). Be sure that each thread instantiates
it's own SSL_CTX and certificate/private key.
Lastly, add some debugs to count the locks by lock ID in your lock handler.
There are about 40 different lock IDs defined in crypto/crypto.h. You'll
need to identify which lock is causing the thread contention. Once you know
that, then you can look in the code to see where that lock is used and
troubleshoot from there.
On 04/23/2015 07:23 PM, Bryan Call wrote:
I was running a benchmark on a 28 core (56 hyper-threaded) server that is
running 84 threads in the process and I am seeing a lot of lock contention.
I saw a lot of lock contention in the calls to SSL_get_error()
ssl3_accept(). I am running RHEL 6.5 and openssl-1.0.1e-30.el6_6.7.x86_64.
We have used SSL_locking_callback() to setup pthread mutex locks, but
haven’t setup dynlocks yet. Would dynlocks help in this scenario?
Output from perf:
Samples: 1M of event 'cycles', Event count (approx.): 602783506611
- 48.04% [ET_NET 0] [kernel.kallsyms] [k] _spin_lock ◆
- _spin_lock ▒
+ 48.05% futex_wait_setup ▒
+ 47.79% futex_wake ▒
+ 1.20% tcp_v4_rcv ▒
+ 0.98% task_rq_lock ▒
+ 0.64% double_lock_balance ▒
+ 3.94% [ET_NET 0] libcrypto.so.1.0.1e [.] bn_mul_mont ▒
+ 1.93% [ET_NET 0] libcrypto.so.1.0.1e [.] BN_usub ▒
+ 1.80% [ET_NET 0] libcrypto.so.1.0.1e [.] 0x000e5200 ▒
+ 1.49% [ET_NET 0] libcrypto.so.1.0.1e [.] sha256_block_data_order ▒
+ 0.97% [ET_NET 0] libcrypto.so.1.0.1e [.] gcm_ghash_clmul ▒
+ 0.85% [ET_NET 0] libpthread-2.12.so [.] pthread_mutex_lock ▒
+ 0.84% [ET_NET 0] libcrypto.so.1.0.1e [.] aesni_ctr32_encrypt_blocks ▒
+ 0.74% [ET_NET 0] libc-2.12.so [.] malloc ▒
+ 0.72% [ET_NET 0] libc-2.12.so [.] memcpy ▒
+ 0.57% [ET_NET 0] libcrypto.so.1.0.1e [.] BN_mod_mul_montgomery ▒
+ 0.51% [ET_NET 0] libpthread-2.12.so [.] pthread_mutex_unlock
pstack of the process (only a couple of the interesting threads):
Thread 1 (Thread 0x7f1fba151800 (LWP 1106)):
#0 0x7f1fbaaeb264 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x7f1fbaae6508 in _L_lock_854 () from /lib64/libpthread.so.0
#2 0x7f1fbaae63d7 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x0072822f in SSL_locking_callback(int, int, char const*, int) ()
#4 0x0031d2ae7f03 in ?? () from /usr/lib64/libcrypto.so.10
#5 0x0031d2ae7d8c in ?? () from /usr/lib64/libcrypto.so.10
#6 0x0031d2ae80ab in ERR_get_state () from /usr/lib64/libcrypto.so.10
#7 0x0031d2ae8557 in ?? () from /usr/lib64/libcrypto.so.10
#8 0x0031d2e424a9 in SSL_get_error () from /usr/lib64/libssl.so.10
#9 0x007249ef in SSLNetVConnection::load_buffer_and_write(long,
long, long, MIOBufferAccessor, int) ()
#10 0x007389df in write_to_net_io(NetHandler*,
Re: [openssl-users] Performance problems with OpenSSL and threading
When you create the private key given to SSL_CTX, be sure to instantiate
a private key for each thread as well. If you share the same key across
all threads, this can introduce contention.
Here are the type 1 2 locks:
# define CRYPTO_LOCK_ERR 1
# define CRYPTO_LOCK_EX_DATA 2
Have you checked the OpenSSL error log to see what errors are being
generated?
Also, if you disable FIPS, does the thread contention still occur?
On 04/24/2015 01:36 PM, Bryan Call wrote:
We are using a single SSL_CTX across all the threads, so I will create
multiple SSL_CTX per thread. I implemented dynlock callbacks
yesterday, but I didn’t see them being used in the tests I did. I
also added atomic counters to see what type of locks are having
contention (see logs below):
Most of the lock contention seems to be around type 2 and type 1 locks
when calling our lock callback. In the FIPS/POST and SSL_CTX
recommendations, in your previous email, what types of locks would
those help with?
Log output:
[Apr 24 10:21:24.339] Server {0xb31} ERROR: SSLUtils.cc
http://SSLUtils.cc:178 (SSL_locking_callback) contention for lock -
total contention: 3938 waiting: 1 file:
/SourceCache/OpenSSL098/OpenSSL098-52.20.2/src/crypto/ex_data.c line:
496 type: 2
[Apr 24 10:21:24.754] Server {0xb10a000} ERROR: SSLUtils.cc
http://SSLUtils.cc:178 (SSL_locking_callback) contention for lock -
total contention: 1350 waiting: 1 file:
/SourceCache/OpenSSL098/OpenSSL098-52.20.2/src/crypto/err/err_def.c
line: 343 type: 1
-Bryan
On Apr 23, 2015, at 4:46 PM, John Foley fol...@cisco.com
mailto:fol...@cisco.com wrote:
Looking at your call stack, it appears you're testing with FIPS mode
enabled. There's a lock in FIPS_selftest_failed() that's used to
ensure the crypto algorithms inside the FIPS boundary are not
utilized until after the POST has completed and passed. Unfortunately
this lock remains installed for the lifetime of the process. We've
seen thread contention in this area when using OpenSSL. One
work-around is to modify your handler that you pass to
SSL_locking_callback() to not lock when the lock ID is
CRYPTO_LOCK_FIPS. But you would only want to disable this particular
lock ID after the POST has completed.
If the above doesn't help, another item to check is whether you're
instantiating an SSL_CTX for each thread. I've seen some developers
use a single SSL_CTX for multiple threads. This results in a single
instance of the server cert and private key being shared across all
the threads. This results in thread contention on the asymmetric key
operations associated with the cert and private key (e.g. RSA). Be
sure that each thread instantiates it's own SSL_CTX and
certificate/private key.
Lastly, add some debugs to count the locks by lock ID in your lock
handler. There are about 40 different lock IDs defined in
crypto/crypto.h. You'll need to identify which lock is causing the
thread contention. Once you know that, then you can look in the code
to see where that lock is used and troubleshoot from there.
On 04/23/2015 07:23 PM, Bryan Call wrote:
I was running a benchmark on a 28 core (56 hyper-threaded) server
that is running 84 threads in the process and I am seeing a lot of
lock contention. I saw a lot of lock contention in the calls to
SSL_get_error() ssl3_accept(). I am running RHEL 6.5
and openssl-1.0.1e-30.el6_6.7.x86_64. We have
used SSL_locking_callback() to setup pthread mutex locks, but
haven’t setup dynlocks yet. Would dynlocks help in this scenario?
Output from perf:
Samples: 1M of event 'cycles', Event count (approx.): 602783506611
- 48.04% [ET_NET 0] [kernel.kallsyms] [k] _spin_lock ◆
- _spin_lock ▒
* + 48.05% futex_wait_setup ▒
* + 47.79% futex_wake ▒
* + 1.20% tcp_v4_rcv ▒
* + 0.98% task_rq_lock ▒
* + 0.64% double_lock_balance ▒
+ 3.94% [ET_NET 0] libcrypto.so.1.0.1e [.] bn_mul_mont ▒
+ 1.93% [ET_NET 0] libcrypto.so.1.0.1e [.] BN_usub ▒
+ 1.80% [ET_NET 0] libcrypto.so.1.0.1e [.] 0x000e5200 ▒
+ 1.49% [ET_NET 0] libcrypto.so.1.0.1e [.] sha256_block_data_order ▒
+ 0.97% [ET_NET 0] libcrypto.so.1.0.1e [.] gcm_ghash_clmul ▒
+ 0.85% [ET_NET 0] libpthread-2.12.so [.] pthread_mutex_lock ▒
+ 0.84% [ET_NET 0] libcrypto.so.1.0.1e [.] aesni_ctr32_encrypt_blocks ▒
+ 0.74% [ET_NET 0] libc-2.12.so [.] malloc ▒
+ 0.72% [ET_NET 0] libc-2.12.so [.] memcpy ▒
+ 0.57% [ET_NET 0] libcrypto.so.1.0.1e [.] BN_mod_mul_montgomery ▒
+ 0.51% [ET_NET 0] libpthread-2.12.so [.] pthread_mutex_unlock
pstack of the process (only a couple of the interesting threads):
Thread 1 (Thread 0x7f1fba151800 (LWP 1106)):
#0 0x7f1fbaaeb264 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x7f1fbaae6508 in _L_lock_854 () from /lib64/libpthread.so.0
#2 0x7f1fbaae63d7 in pthread_mutex_lock () from
/lib64/libpthread.so.0
#3 0x0072822f in SSL_locking_callback(int, int, char const*,
int) ()
#4
Re: [openssl-users] FIPS: SSL 3.0 now forbidden in latest NDCPP update
Hello, In FIPS mode SSL 3.0 is not allowed: that has always been the case. % openssl version OpenSSL 1.0.1f 6 Jan 2014 % OPENSSL_FIPS=1 openssl ciphers -v | grep SSL ECDHE-RSA-AES256-SHASSLv3 ECDHE-ECDSA-AES256-SHA SSLv3 DHE-RSA-AES256-SHA SSLv3 DHE-DSS-AES256-SHA SSLv3 [snipped] All of the others are TLSv1.2. Why is SSLv3.0 seen in FIPS mode on this install ? TLS 1.0 is currently permitted though. As far as OpenSSL is concerned, will any action to remove TLSv.10 only be taken when put into a FIPS frame, eg. the recent NDCPPv1.0 is not enough ground to make a change ? Regards. -- View this message in context: http://openssl.6102.n7.nabble.com/FIPS-SSL-3-0-now-forbidden-in-latest-NDCPP-update-tp57695p57707.html Sent from the OpenSSL - User mailing list archive at Nabble.com. ___ openssl-users mailing list To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-users
Re: [openssl-users] Performance problems with OpenSSL and threading
In my last email I ran the benchmark on Fedora 21 (big mistake). Here are the
results when running it back on the 28/56 core RHEL 6.5 server showing
contention in a different place (fips). Is there a reason it would be calling
into the FIPS code to get a lock of FIPS is not enabled?
Contention log:
[Apr 24 21:35:09.731] Server {0x7f5529ccd700} ERROR: contention for lock -
total contention: 70662 waiting: 16 file: fips.c line: 471 type: 40
[Apr 24 21:35:09.732] Server {0x7f55299ca700} ERROR: contention for lock -
total contention: 71605 waiting: 2 file: fips.c line: 107 type: 39
Line in in fips that see the contention the most:
[bcall@bm77 trafficserver]$ grep fips.c diags.log | cut -f 19 -d ' ' | sort |
uniq -c | sort -rn
875379 471
288338 107
275472 127
Different lock types and their contention frequency (multiple benchmarks):
875379 40 - in fips.c
563810 39 - also in fips.c
440518 1
44145 18
17447 2
11356 10
1377 19
687 12
532 25
302 33
239 9
58 16
40 21
32 11
19 3
1 14
Code to diable FIPS and log:
#ifdef OPENSSL_FIPS
int mode = FIPS_mode();
Debug(ssl, FIPS_mode: %d, mode);
mode = FIPS_mode();
if (mode) {
FIPS_mode_set(0);
Debug(ssl, FIPS_mode: %d, mode);
}
#endif
[Apr 24 21:43:45.860] Server {0x7f7628146800} DEBUG: (ssl) FIPS_mode: 0
-Bryan
On Apr 24, 2015, at 10:56 AM, John Foley fol...@cisco.com
mailto:fol...@cisco.com wrote:
When you create the private key given to SSL_CTX, be sure to instantiate a
private key for each thread as well. If you share the same key across all
threads, this can introduce contention.
Here are the type 1 2 locks:
# define CRYPTO_LOCK_ERR 1
# define CRYPTO_LOCK_EX_DATA 2
Have you checked the OpenSSL error log to see what errors are being generated?
Also, if you disable FIPS, does the thread contention still occur?
On 04/24/2015 01:36 PM, Bryan Call wrote:
We are using a single SSL_CTX across all the threads, so I will create
multiple SSL_CTX per thread. I implemented dynlock callbacks yesterday, but
I didn’t see them being used in the tests I did. I also added atomic
counters to see what type of locks are having contention (see logs below):
Most of the lock contention seems to be around type 2 and type 1 locks when
calling our lock callback. In the FIPS/POST and SSL_CTX recommendations, in
your previous email, what types of locks would those help with?
Log output:
[Apr 24 10:21:24.339] Server {0xb31} ERROR: SSLUtils.cc
http://sslutils.cc/:178 (SSL_locking_callback) contention for lock -
total contention: 3938 waiting: 1 file:
/SourceCache/OpenSSL098/OpenSSL098-52.20.2/src/crypto/ex_data.c line: 496
type: 2
[Apr 24 10:21:24.754] Server {0xb10a000} ERROR: SSLUtils.cc
http://sslutils.cc/:178 (SSL_locking_callback) contention for lock -
total contention: 1350 waiting: 1 file:
/SourceCache/OpenSSL098/OpenSSL098-52.20.2/src/crypto/err/err_def.c line:
343 type: 1
-Bryan
On Apr 23, 2015, at 4:46 PM, John Foley fol...@cisco.com
mailto:fol...@cisco.com wrote:
Looking at your call stack, it appears you're testing with FIPS mode
enabled. There's a lock in FIPS_selftest_failed() that's used to ensure
the crypto algorithms inside the FIPS boundary are not utilized until after
the POST has completed and passed. Unfortunately this lock remains
installed for the lifetime of the process. We've seen thread contention in
this area when using OpenSSL. One work-around is to modify your handler
that you pass to SSL_locking_callback() to not lock when the lock ID is
CRYPTO_LOCK_FIPS. But you would only want to disable this particular lock
ID after the POST has completed.
If the above doesn't help, another item to check is whether you're
instantiating an SSL_CTX for each thread. I've seen some developers use a
single SSL_CTX for multiple threads. This results in a single instance of
the server cert and private key being shared across all the threads. This
results in thread contention on the asymmetric key operations associated
with the cert and private key (e.g. RSA). Be sure that each thread
instantiates it's own SSL_CTX and certificate/private key.
Lastly, add some debugs to count the locks by lock ID in your lock handler.
There are about 40 different lock IDs defined in crypto/crypto.h. You'll
need to identify which lock is causing the thread contention. Once you
know that, then you can look in the code to see where that lock is used and
troubleshoot from there.
On 04/23/2015 07:23 PM, Bryan Call wrote:
I was running a benchmark on a 28 core (56 hyper-threaded) server that is
running 84 threads in the process and I am seeing a lot of lock
contention. I saw a lot of lock contention in the calls to
SSL_get_error() ssl3_accept(). I am running RHEL 6.5 and
Re: [openssl-users] FIPS: SSL 3.0 now forbidden in latest NDCPP update
On Fri, Apr 24, 2015, jonetsu wrote: Hello, In FIPS mode SSL 3.0 is not allowed: that has always been the case. % openssl version OpenSSL 1.0.1f 6 Jan 2014 % OPENSSL_FIPS=1 openssl ciphers -v | grep SSL ECDHE-RSA-AES256-SHASSLv3 ECDHE-ECDSA-AES256-SHA SSLv3 DHE-RSA-AES256-SHA SSLv3 DHE-DSS-AES256-SHA SSLv3 [snipped] All of the others are TLSv1.2. Why is SSLv3.0 seen in FIPS mode on this install ? That refers to the minimum version of the ciphersuite: it doesn't imply that it will only be used in SSLv3 (which is disabled in FIPS mode). Steve. -- Dr Stephen N. Henson. OpenSSL project core developer. Commercial tech support now available see: http://www.openssl.org ___ openssl-users mailing list To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-users
