[Cryptodev-linux-devel] Problem with OpenSSH/OpenSSL Interaction When Cryptodev is Used
I am experiencing a rather odd problem with cryptodev that I am not sure where to follow next. The distro is RedSleeve (EL6 armv5tel). OpenSSH 5.3p1 (EL6 extra patches) OpenSSL 1.0.1e (EL6 extra patches), rebuild with -DHAVE_CRYPTODEV (without digests) Cryptodev 0.9 (yes, I know this is ancient but I tried 1.7 and it made no difference) Linux 3.4.6 (old, but heavily patched for both the hardware compatibility and CVEs, binary that ships with my device) What works: mod_ssl with ECDHE-RSA-AES128-GCM-SHA256 openssl s_server <-> s_client What doesn't: sshd with offloadable ciphers Non-offloadable ciphers (e.g. blowfish) always work fine. aes128-cbc works without cryptodev module loaded, but when it is loaded the connection breaks _after_ sending the last login information. With cryptodev loaded and Ciphers set to aes128-cbc: On the client side, everything seems to work fine, right up to the end: $ ssh localhost [...] debug2: PTY allocation request accepted on channel 0 debug2: channel 0: rcvd adjust 2097152 debug2: channel_input_status_confirm: type 99 id 0 debug2: shell request accepted on channel 0 Last login: Tue May 26 16:40:17 2015 from localhost debug1: channel 0: free: client-session, nchannels 1 debug3: channel 0: status: The following connections are open: #0 client-session (t4 r0 i0/0 o0/0 fd 4/5 cc -1) Connection to qnap closed by remote host. Connection to qnap closed. Transferred: sent 3472, received 4128 bytes, in 0.5 seconds Bytes per second: sent 7622.1, received 9062.3 debug1: Exit status -1 This tells me that the connection setup actually succeeds, since this gets received by the client: Last login: Tue May 26 16:40:17 2015 from localhost On the server side with LogLevel DEBUG: Everything looks OK initially: [...] May 26 16:59:49 qnap sshd[29520]: debug1: Allocating pty. May 26 16:59:49 qnap sshd[29474]: debug1: session_new: session 0 May 26 16:59:49 qnap sshd[29520]: debug1: session_pty_req: session 0 alloc /dev/pts/4 May 26 16:59:49 qnap sshd[29520]: debug1: server_input_channel_req: channel 0 request env reply 0 May 26 16:59:49 qnap sshd[29520]: debug1: session_by_channel: session 0 channel 0 May 26 16:59:49 qnap sshd[29520]: debug1: session_input_channel_req: session 0 req env May 26 16:59:49 qnap sshd[29520]: debug1: server_input_channel_req: channel 0 request env reply 0 May 26 16:59:49 qnap sshd[29520]: debug1: session_by_channel: session 0 channel 0 May 26 16:59:49 qnap sshd[29520]: debug1: session_input_channel_req: session 0 req env May 26 16:59:49 qnap sshd[29520]: debug1: server_input_channel_req: channel 0 request shell reply 1 May 26 16:59:49 qnap sshd[29520]: debug1: session_by_channel: session 0 channel 0 May 26 16:59:49 qnap sshd[29520]: debug1: session_input_channel_req: session 0 req shell May 26 16:59:49 qnap sshd[29521]: debug1: Setting controlling tty using TIOCSCTTY. Right up to here: May 26 16:59:49 qnap sshd[29520]: fatal: evp_crypt: EVP_Cipher failed The rest appears to be just clean-up: May 26 16:59:49 qnap sshd[29520]: debug1: do_cleanup May 26 16:59:49 qnap sshd[29474]: debug1: do_cleanup May 26 16:59:49 qnap sshd[29474]: debug1: PAM: cleanup May 26 16:59:49 qnap sshd[29474]: debug1: PAM: closing session May 26 16:59:49 qnap sshd[29474]: pam_unix(sshd:session): session closed for user gordan May 26 16:59:49 qnap sshd[29474]: debug1: PAM: deleting credentials May 26 16:59:49 qnap sshd[29474]: debug1: session_pty_cleanup: session 0 release /dev/pts/4 The error: sshd[29520]: fatal: evp_crypt: EVP_Cipher failed appears to be emitted by sshd rather than OpenSSL or Cryptodev. The relevant bit of code is in OpenSSH's cipher.c file: void cipher_crypt(CipherContext *cc, u_char *dest, const u_char *src, u_int len) { if (len % cc->cipher->block_size) fatal("cipher_encrypt: bad plaintext length %d", len); if (EVP_Cipher(&cc->evp, dest, (u_char *)src, len) == 0) fatal("evp_crypt: EVP_Cipher failed"); } EVP_Cipher function is part of OpenSSL, declared in: crypto/evp/evp.h: int EVP_Cipher(EVP_CIPHER_CTX *c, unsigned char *out, const unsigned char *in, unsigned int inl); and defined in: crypto/evp/evp_lib.c: int EVP_Cipher(EVP_CIPHER_CTX *ctx, unsigned char *out, const unsigned char *in, unsigned int inl) { #ifdef OPENSSL_FIPS FIPS_selftest_check(); #endif return ctx->cipher->do_cipher(ctx,out,in,inl); } So it would appear that ctx->cipher->do_cipher(ctx,out,in,inl); returns 0. Any advice on the best way to proceed with troubleshooting this would be greatly appreciated. Many thanks. Gordan ___ Cryptodev-linux-devel mailing list Cryptodev-linux-devel@gna.org https://mail.gna.org/listinfo/cryptodev-linux-devel
Re: [Cryptodev-linux-devel] Problem with OpenSSH/OpenSSL Interaction When Cryptodev is Used
On 2015-05-26 17:05, Gordan Bobic wrote: I am experiencing a rather odd problem with cryptodev that I am not sure where to follow next. The distro is RedSleeve (EL6 armv5tel). OpenSSH 5.3p1 (EL6 extra patches) OpenSSL 1.0.1e (EL6 extra patches), rebuild with -DHAVE_CRYPTODEV (without digests) Cryptodev 0.9 (yes, I know this is ancient but I tried 1.7 and it made no difference) Linux 3.4.6 (old, but heavily patched for both the hardware compatibility and CVEs, binary that ships with my device) What works: mod_ssl with ECDHE-RSA-AES128-GCM-SHA256 openssl s_server <-> s_client What doesn't: sshd with offloadable ciphers Non-offloadable ciphers (e.g. blowfish) always work fine. aes128-cbc works without cryptodev module loaded, but when it is loaded the connection breaks _after_ sending the last login information. With cryptodev loaded and Ciphers set to aes128-cbc: On the client side, everything seems to work fine, right up to the end: $ ssh localhost [...] debug2: PTY allocation request accepted on channel 0 debug2: channel 0: rcvd adjust 2097152 debug2: channel_input_status_confirm: type 99 id 0 debug2: shell request accepted on channel 0 Last login: Tue May 26 16:40:17 2015 from localhost debug1: channel 0: free: client-session, nchannels 1 debug3: channel 0: status: The following connections are open: #0 client-session (t4 r0 i0/0 o0/0 fd 4/5 cc -1) Connection to qnap closed by remote host. Connection to qnap closed. Transferred: sent 3472, received 4128 bytes, in 0.5 seconds Bytes per second: sent 7622.1, received 9062.3 debug1: Exit status -1 This tells me that the connection setup actually succeeds, since this gets received by the client: Last login: Tue May 26 16:40:17 2015 from localhost On the server side with LogLevel DEBUG: Everything looks OK initially: [...] May 26 16:59:49 qnap sshd[29520]: debug1: Allocating pty. May 26 16:59:49 qnap sshd[29474]: debug1: session_new: session 0 May 26 16:59:49 qnap sshd[29520]: debug1: session_pty_req: session 0 alloc /dev/pts/4 May 26 16:59:49 qnap sshd[29520]: debug1: server_input_channel_req: channel 0 request env reply 0 May 26 16:59:49 qnap sshd[29520]: debug1: session_by_channel: session 0 channel 0 May 26 16:59:49 qnap sshd[29520]: debug1: session_input_channel_req: session 0 req env May 26 16:59:49 qnap sshd[29520]: debug1: server_input_channel_req: channel 0 request env reply 0 May 26 16:59:49 qnap sshd[29520]: debug1: session_by_channel: session 0 channel 0 May 26 16:59:49 qnap sshd[29520]: debug1: session_input_channel_req: session 0 req env May 26 16:59:49 qnap sshd[29520]: debug1: server_input_channel_req: channel 0 request shell reply 1 May 26 16:59:49 qnap sshd[29520]: debug1: session_by_channel: session 0 channel 0 May 26 16:59:49 qnap sshd[29520]: debug1: session_input_channel_req: session 0 req shell May 26 16:59:49 qnap sshd[29521]: debug1: Setting controlling tty using TIOCSCTTY. Right up to here: May 26 16:59:49 qnap sshd[29520]: fatal: evp_crypt: EVP_Cipher failed The rest appears to be just clean-up: May 26 16:59:49 qnap sshd[29520]: debug1: do_cleanup May 26 16:59:49 qnap sshd[29474]: debug1: do_cleanup May 26 16:59:49 qnap sshd[29474]: debug1: PAM: cleanup May 26 16:59:49 qnap sshd[29474]: debug1: PAM: closing session May 26 16:59:49 qnap sshd[29474]: pam_unix(sshd:session): session closed for user gordan May 26 16:59:49 qnap sshd[29474]: debug1: PAM: deleting credentials May 26 16:59:49 qnap sshd[29474]: debug1: session_pty_cleanup: session 0 release /dev/pts/4 The error: sshd[29520]: fatal: evp_crypt: EVP_Cipher failed appears to be emitted by sshd rather than OpenSSL or Cryptodev. The relevant bit of code is in OpenSSH's cipher.c file: void cipher_crypt(CipherContext *cc, u_char *dest, const u_char *src, u_int len) { if (len % cc->cipher->block_size) fatal("cipher_encrypt: bad plaintext length %d", len); if (EVP_Cipher(&cc->evp, dest, (u_char *)src, len) == 0) fatal("evp_crypt: EVP_Cipher failed"); } EVP_Cipher function is part of OpenSSL, declared in: crypto/evp/evp.h: int EVP_Cipher(EVP_CIPHER_CTX *c, unsigned char *out, const unsigned char *in, unsigned int inl); and defined in: crypto/evp/evp_lib.c: int EVP_Cipher(EVP_CIPHER_CTX *ctx, unsigned char *out, const unsigned char *in, unsigned int inl) { #ifdef OPENSSL_FIPS FIPS_selftest_check(); #endif return ctx->cipher->do_cipher(ctx,out,in,inl); } So it would appear that ctx->cipher->do_cipher(ctx,out,in,inl); returns 0. A bit of extra info, with cryptodev_verbosity=2, on 0.9 when the error occurs: cryptodev: sshd[1205]: invalid session ID=0xAADBA6A0 With 1.7: cryptodev: sshd[1520] (fill_kcop_from_cop:647): invalid session ID=0xEC91F39A cryptodev: sshd[1520] (cryptodev_ioctl:857): Error copying from user ___ Cryptodev-linux-devel mailing list Cryptodev-linux-devel@gna.o
Re: [Cryptodev-linux-devel] Problem with OpenSSH/OpenSSL Interaction When Cryptodev is Used
Hi, On Tue, May 26, 2015 at 07:29:34PM +0100, Gordan Bobic wrote: > A bit of extra info, with cryptodev_verbosity=2, on 0.9 when the error > occurs: > cryptodev: sshd[1205]: invalid session ID=0xAADBA6A0 > > With 1.7: > cryptodev: sshd[1520] (fill_kcop_from_cop:647): invalid session > ID=0xEC91F39A > cryptodev: sshd[1520] (cryptodev_ioctl:857): Error copying from user But the test code (i.e. the various *_comp programs) succeeds? I really should take a look at the openssl integration code. Last time I did, the mere fact it always uses a ridiculously small (4k or so) buffer was enough for me. Cheers, Phil ___ Cryptodev-linux-devel mailing list Cryptodev-linux-devel@gna.org https://mail.gna.org/listinfo/cryptodev-linux-devel