Oliver Graute wrote: > On 25/02/16, Rob Crittenden wrote: >> Oliver Graute wrote: >>> On 25/02/16, Oliver Graute wrote: >>>> On 24/02/16, Rob Crittenden wrote: >>>>> Oliver Graute wrote: >>>>>> On 23/02/16, Oliver Graute wrote: >>>>>>> On Tue, Feb 23, 2016 at 5:10 PM, Oliver Graute >>>>>>> <[email protected]> wrote: >>>>>>>> On 23/02/16, Rob Crittenden wrote: >>>>>>>>> Oliver Graute wrote: >>>>>>>>>> On 22/02/16, Rob Crittenden wrote: >>>>>>>>>>> Oliver Graute wrote: >>>>>>>>>>>> Hello, >>>>>>>>>>>> >>>>>>>>>>>> I installed the mod_nss plugin in version 1.0.12 on my apache >>>>>>>>>>>> webserver, >>>>>>>>>>>> TLS on Port 443 is working fine until I enable the new NSSSession >>>>>>>>>>>> ticket >>>>>>>>>>>> feature in my nss.conf with: >>>>>>>>>>>> >>>>>>>>>>>> #RFC 5077 >>>>>>>>>>>> NSSSessionTickets on >>>>>>>>>>>> >>>>>>>>>>>> then something is broken, I see segfaults in my apache error log: >>>>>>>>>>>> >>>>>>>>>>>> [Fri Feb 19 10:12:15.338660 2016] [mpm_prefork:notice] [pid 413] >>>>>>>>>>>> AH00163: Apache/2.4.16 (Unix) mod_nss/1.0.12 NSS/3.19.2 Basic ECC >>>>>>>>>>>> PHP/5.5.10 configured -- resuming normal operations >>>>>>>>>>>> [Fri Feb 19 10:12:15.338843 2016] [mpm_prefork:info] [pid 413] >>>>>>>>>>>> AH00164: Server built: Feb 22 2016 12:44:38 >>>>>>>>>>>> [Fri Feb 19 10:12:15.339046 2016] [core:notice] [pid 413] AH00094: >>>>>>>>>>>> Command line: '/usr/sbin/httpd -D FOREGROUND -D SSL -D PHP5' >>>>>>>>>>>> [Fri Feb 19 10:12:15.339160 2016] [mpm_prefork:debug] [pid 413] >>>>>>>>>>>> prefork.c(995): AH00165: Accept mutex: sysvsem (default: sysvsem) >>>>>>>>>>>> [Fri Feb 19 10:12:15.386483 2016] [:debug] [pid 416] >>>>>>>>>>>> nss_engine_init.c(286): SNI is enabled >>>>>>>>>>>> [Fri Feb 19 10:12:15.386853 2016] [:info] [pid 416] Init: Seeding >>>>>>>>>>>> PRNG with 136 bytes of entropy >>>>>>>>>>>> [Fri Feb 19 10:12:40.374175 2016] [core:notice] [pid 413] AH00052: >>>>>>>>>>>> child pid 416 exit signal Segmentation fault (11) >>>>>>>>>>>> [Fri Feb 19 10:12:41.496820 2016] [:debug] [pid 423] >>>>>>>>>>>> nss_engine_init.c(286): SNI is enabled >>>>>>>>>>>> [Fri Feb 19 10:12:41.497224 2016] [:info] [pid 423] Init: Seeding >>>>>>>>>>>> PRNG with 136 bytes of entropy >>>>>>>>>>>> [Fri Feb 19 10:12:42.388948 2016] [core:notice] [pid 413] AH00052: >>>>>>>>>>>> child pid 423 exit signal Segmentation fault (11) >>>>>>>>>>>> [Fri Feb 19 10:12:43.508779 2016] [:debug] [pid 424] >>>>>>>>>>>> nss_engine_init.c(286): SNI is enabled >>>>>>>>>>>> [Fri Feb 19 10:12:43.509217 2016] [:info] [pid 424] Init: Seeding >>>>>>>>>>>> PRNG with 136 bytes of entropy >>>>>>>>>>>> [Fri Feb 19 10:12:44.404130 2016] [core:notice] [pid 413] AH00052: >>>>>>>>>>>> child pid 424 exit signal Segmentation fault (11) >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> and in Chrome Browser I got: >>>>>>>>>>>> >>>>>>>>>>>> ERR_SSL_VERSION_OR_CIPHER_MISMATCH >>>>>>>>>>>> >>>>>>>>>>>> I tested also a basic ssl client connection with openssl: >>>>>>>>>>>> >>>>>>>>>>>> openssl s_client -connect 192.168.1.229:443 -state -debug >>>>>>>>>>>> >>>>>>>>>>>> SSL_connect:SSLv3 read server certificate A >>>>>>>>>>>> SSL_connect:SSLv3 read server key exchange A >>>>>>>>>>>> SSL_connect:SSLv3 read server done A >>>>>>>>>>>> write to 0x205dec0 [0x206dd50] (75 bytes => 75 (0x4B)) >>>>>>>>>>>> 0000 - 16 03 03 00 46 10 00 00-42 41 04 3d c7 93 63 45 >>>>>>>>>>>> ....F...BA.=..cE >>>>>>>>>>>> 0010 - 79 41 11 bc 06 c0 b7 c6-d1 b5 33 d9 86 a6 d5 e9 >>>>>>>>>>>> yA........3..... >>>>>>>>>>>> 0020 - 36 e4 2b ac 0e bc 70 d6-d6 8c a7 a9 3c dd 1b 0c >>>>>>>>>>>> 6.+...p.....<... >>>>>>>>>>>> 0030 - 77 48 20 38 dd 1e c9 a1-05 6c 5c b6 c9 f4 99 f2 wH >>>>>>>>>>>> 8.....l\..... >>>>>>>>>>>> 0040 - 1a 18 ae 81 63 71 65 90-e8 a5 b6 >>>>>>>>>>>> ....cqe.... >>>>>>>>>>>> SSL_connect:SSLv3 write client key exchange A >>>>>>>>>>>> write to 0x205dec0 [0x206dd50] (6 bytes => 6 (0x6)) >>>>>>>>>>>> 0000 - 14 03 03 00 01 01 ...... >>>>>>>>>>>> SSL_connect:SSLv3 write change cipher spec A >>>>>>>>>>>> write to 0x205dec0 [0x206dd50] (45 bytes => 45 (0x2D)) >>>>>>>>>>>> 0000 - 16 03 03 00 28 b1 e0 60-8a 2c 97 cf a0 4f 97 ee >>>>>>>>>>>> ....(..`.,...O.. >>>>>>>>>>>> 0010 - cd 8f 05 41 aa 50 a6 73-a3 4c 86 1e 5f 3c 7b 2b >>>>>>>>>>>> ...A.P.s.L.._<{+ >>>>>>>>>>>> 0020 - 2d 7e 6a 68 dc 97 94 9d-91 15 c0 0e 27 >>>>>>>>>>>> -~jh........' >>>>>>>>>>>> SSL_connect:SSLv3 write finished A >>>>>>>>>>>> SSL_connect:SSLv3 flush data >>>>>>>>>>>> read from 0x205dec0 [0x2063f83] (5 bytes => 0 (0x0)) >>>>>>>>>>>> SSL_connect:failed in SSLv3 read server session ticket A >>>>>>>>>>>> 140123095688864:error:140790E5:SSL routines:SSL23_WRITE:ssl >>>>>>>>>>>> handshake failure:s23_lib.c:177: >>>>>>>>>>>> >>>>>>>>>>>> apache and mod_nss are build from the sources for an embedded >>>>>>>>>>>> yocto environment. >>>>>>>>>>>> >>>>>>>>>>>> some ideas, whats going on here? >>>>>>>>>>> >>>>>>>>>>> Can you get a stack trace from the core? >>>>>>>>>> >>>>>>>>>> I can give you an strace, see below. Other stack tools are currently >>>>>>>>>> not >>>>>>>>>> available, because I need to compile them first for my yocto >>>>>>>>>> environment. If you need something special please tell me. >>>>>>>>>> >>>>>>>>>>> This is Apache 2.4.x? >>>>>>>>>> >>>>>>>>>> yes it is Apache 2.4.16 >>>>>>>>>> >>>>>>>>>>> Is it failing on a request or on startup? >>>>>>>>>> >>>>>>>>>> its failing on every https request. >>>>>>>>> >>>>>>>>> strace in this case isn't particularly helpful as it doesn't show >>>>>>>>> where >>>>>>>>> it is crashing. >>>>>>>>> >>>>>>>>> Can I see your nss.conf? >>>>>>>>> >>>>>>>>> What version of NSS are you using? >>>>>>>> >>>>>>>> I'am using nss in version 3.19.2 >>>>>>>> >>>>>>>> here my nss.conf >>>>>>>> >>>>>>>> # >>>>>>>> # This is the Apache server configuration file providing SSL support >>>>>>>> using. >>>>>>>> # the mod_nss plugin. It contains the configuration directives to >>>>>>>> instruct >>>>>>>> # the server how to serve pages over an https connection. >>>>>>>> # >>>>>>>> # Do NOT simply read the instructions in here without understanding >>>>>>>> # what they do. They're here only as hints or reminders. If you are >>>>>>>> unsure >>>>>>>> # consult the online docs. You have been warned. >>>>>>>> # >>>>>>>> >>>>>>>> # >>>>>>>> # When we also provide SSL we have to listen to the >>>>>>>> # standard HTTP port (see above) and to the HTTPS port >>>>>>>> # >>>>>>>> # Note: Configurations that use IPv6 but not IPv4-mapped addresses >>>>>>>> need two >>>>>>>> # Listen directives: "Listen [::]:443" and "Listen 0.0.0.0:443" >>>>>>>> # >>>>>>>> Listen 443 >>>>>>>> >>>>>>>> ## >>>>>>>> ## SSL Global Context >>>>>>>> ## >>>>>>>> ## All SSL configuration in this context applies both to >>>>>>>> ## the main server and all SSL-enabled virtual hosts. >>>>>>>> ## >>>>>>>> >>>>>>>> # >>>>>>>> # Some MIME-types for downloading Certificates and CRLs >>>>>>>> # >>>>>>>> AddType application/x-x509-ca-cert .crt >>>>>>>> AddType application/x-pkcs7-crl .crl >>>>>>>> >>>>>>>> >>>>>>>> # Pass Phrase Helper: >>>>>>>> # This helper program stores the token password pins between >>>>>>>> # restarts of Apache. >>>>>>>> # Unfortunately the directive is required even if we use no password >>>>>>>> # (though in such case the program should never be used) >>>>>>>> NSSPassPhraseHelper /usr/lib/apache2/bin/nss_pcache >>>>>>>> >>>>>>>> # Pass Phrase Dialog: >>>>>>>> # Configure the pass phrase gathering process. >>>>>>>> # The filtering dialog program (`builtin' is a internal >>>>>>>> # terminal dialog) has to provide the pass phrase on stdout. >>>>>>>> #NSSPassPhraseDialog builtin >>>>>>>> NSSPassPhraseDialog file:/etc/apache2/password.conf >>>>>>>> >>>>>>>> # Configure the SSL Session Cache. >>>>>>>> # NSSSessionCacheSize is the number of entries in the cache. >>>>>>>> # NSSSessionCacheTimeout is the SSL2 session timeout (in seconds). >>>>>>>> # NSSSession3CacheTimeout is the SSL3/TLS session timeout (in >>>>>>>> seconds). >>>>>>>> NSSSessionCacheSize 10000 >>>>>>>> NSSSessionCacheTimeout 100 >>>>>>>> NSSSession3CacheTimeout 86400 >>>>>>>> >>>>>>>> #RFC 5077 >>>>>>>> NSSSessionTickets off >>>>>>> >>>>>>> SessionTickets are on not off >>>>>>> >>>>>>> NSSSessionTickets on >>>>>>> >>>>>> >>>>>> in my /etc/apache2/extra/httpd-vhosts.conf I also added some mod_nss >>>>>> parameters for every virtual host. It seems that the problem is here. >>>>>> Because >>>>>> if I remove the include of httpd-vhosts.conf no segfaults occur. >>>>>> >>>>>> here my httpd-vhosts.conf with nss related parameters. Are these >>>>>> parameters capable for a virtual host configuration? >>>>> >>>>> You've included some directives that are expected to be global: >>>>> >>>>> NSSPassPhraseHelper >>>>> NSSPassPhraseDialog >>>>> NSSSession*Cache* >>>>> NSSCertificateDatabase >>>>> >>>>> I doubt they hurt anything. >>>> >>>> ok thx, I removed these directives from the httpd-vhosts.conf >>>> >>>>> >>>>> I still can't duplicate a crash. I ran it through valgrind to see if >>>>> there was a memory issue and came up with nothing too. >>>>> >>>>> This could be an architecture problem that I just can't see on x86 >>>>> hardware I suppose. >>>> >>>> I'am using a arm hardware (imx6) >>>> >>>>> This particular crash seems rather strange too. Within mod_nss enabling >>>>> this just calls: >>>>> >>>>> SSL_OptionSet(mctx->model, SSL_ENABLE_SESSION_TICKETS, >>>>> mctx->sc->session_tickets); >>>>> >>>>> Basically enabling it in the model server socket. I would imagine the >>>>> crash is probably deeper in NSS. >>>> >>>> >>>>> To narrow things down you might try the NSS tool selfserv. It has an >>>>> option, -u, to enable session tickets. It might eliminate mod_nss as the >>>>> crash source (or it might implicate it too). >>>> >>>> >>>> thx for the hint to the NSS selfserv tool. >>>> >>>> I'am using it with DSA Keys instead of Eliptic Curve Keys because >>>> selfserf can't handle ECC. (selfserv -Y didn't show me ECC ciphers) >>>> >>>> First I generate a new nss db. >>>> >>>> certutil -N -d /etc/apache2/nss-conf -f /etc/apache2/password2.conf >>>> certutil -G -k dsa -n localhost -t "TC,," -d /etc/apache2/nss-conf -f >>>> /etc/apache2/password2.conf >>>> certutil -K -d /etc/apache2/nss-conf -f /etc/apache2/password2.conf >>>> certutil -L -d /etc/apache2/nss-conf -f /etc/apache2/password2.conf >>>> certutil -S -s "CN=localhost" -x -n localhost -t "TC,," -d >>>> /etc/apache2/nss-conf/ -f /etc/apache2/password2.conf >>>> >>>> Then starting selfserv with params: >>>> >>>> selfserv -n "localhost" -V tls1.2: -d /etc/apache2/nss-conf/ -f >>>> /etc/apache2/password2.conf -p 443 -v -u >>>> >>>> selfserv: About to call accept. >>>> >>>> Now I perform a https request with the Chrome Browser >>>> >>>> GET /test.php HTTP/1.1 >>>> Host: 192.168.1.229 >>>> Connection: keep-alive >>>> Cache-Control: max-age=0 >>>> Accept: >>>> text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8 >>>> Upgrade-Insecure-Requests: 1 >>>> User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, >>>> like Gecko) Chrome/46.0.2490.80 Safari/537.36 >>>> DNT: 1 >>>> Accept-Encoding: gzip, deflate, sdch >>>> Accept-Language: de-DE,de;q=0.8,en-US;q=0.6,en;q=0.4 >>>> >>>> EOF >>>> >>>> it looks for me that it works with selfserv. But I'm not sure if the >>>> Session >>>> Ticket works and if it makes a difference not using ECC Keys. >>> >>> I checked with wireshark, and I saw the TLS Session Ticket Extension >>> in the hello packet there. >>> >>> I'also switched back to my old ciphers in my apache setup nss.conf >>> >>> NSSCipherSuite >>> +rsa_rc4_128_md5,+rsa_rc4_128_sha,+rsa_3des_sha,-rsa_des_sha,-rsa_rc4_40_md5,-rsa_rc2_40_md5,-rsa_null_md5,-rsa_null_sha,+fips_3des_sha,-fips_des_sha,-rsa_des_56_sha,-rsa_rc4_56_sha,+rsa_aes_128_sha,+rsa_aes_256_sha >>> >>> and this is working together with NSSSessionTickets on. >>> >>> So it breaks if i'am using ECC Ciphersuite together with Session Tickets! >>> >>> NSSCipherSuite +ecdhe_ecdsa_aes_128_gcm_sha256,+ecdhe_ecdsa_aes_128_sha256 >>> NSSSessionTickets on >>> >>> is this something that can be explained? >> >> Hmm, interesting. It gives me something to look at. I was in the middle >> of responding to your previous message when this one came in. >> >> I had been testing with an RSA key. Let me try an ECC key to see if I >> can reproduce the crash (no success yet). >> >> Here is the previous bit I was going to send: >> >> This actually generated an RSA key for the cert you used for selfserv. >> certutil will generate a certificate if the -k <id> option isn't provided. >> >> You can do the same with ECC by adding to the last command -k ec -q <curve> > > > I updated from nss 3.19 to nss 3.21 but I run into the same issue with it. > > Then I played around with gdb and saw that it breaks in libssl3.so > Some hints how I can debug this further? > > Best Regards, > > Oliver > > gdb httpd > run -X -e debug -D /usr/sbin/httpd > > root@ibis-box:/var/apache2/logs# gdb httpd > GNU gdb (GDB) 7.9.1 > Copyright (C) 2015 Free Software Foundation, Inc. > License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> > This is free software: you are free to change and redistribute it. > There is NO WARRANTY, to the extent permitted by law. Type "show copying" > and "show warranty" for details. > This GDB was configured as "arm-poky-linux-gnueabi". > Type "show configuration" for configuration details. > For bug reporting instructions, please see: > <http://www.gnu.org/software/gdb/bugs/>. > Find the GDB manual and other documentation resources online at: > <http://www.gnu.org/software/gdb/documentation/>. > For help, type "help". > Type "apropos word" to search for commands related to "word"... > Reading symbols from httpd...(no debugging symbols found)...done. > (gdb) run -X -e debug -D /usr/sbin/httpd > Starting program: /usr/sbin/httpd -X -e debug -D /usr/sbin/httpd > warning: File "/lib/libthread_db-1.0.so" auto-loading has been declined by > your `auto-load safe-path' set to "$debugdir:$datadir/auto-load". > To enable execution of this file add > add-auto-load-safe-path /lib/libthread_db-1.0.so > line to your configuration file "/home/root/.gdbinit". > To completely disable this security protection add > set auto-load safe-path / > line to your configuration file "/home/root/.gdbinit". > For more information about this security protection see the > "Auto-loading safe path" section in the GDB manual. E.g., run from the shell: > info "(gdb)Auto-loading safe path" > warning: Unable to find libthread_db matching inferior's thread library, > thread debugging will not be available. > [Fri Feb 26 00:03:18.648483 2016] [so:debug] [pid 441] mod_so.c(266): > AH01575: loaded module authn_file_module from > /usr/lib/apache2/modules/mod_authn_file.so > [Fri Feb 26 00:03:18.700565 2016] [so:debug] [pid 441] mod_so.c(266): > AH01575: loaded module authn_core_module from > /usr/lib/apache2/modules/mod_authn_core.so > [Fri Feb 26 00:03:18.750258 2016] [so:debug] [pid 441] mod_so.c(266): > AH01575: loaded module authz_host_module from > /usr/lib/apache2/modules/mod_authz_host.so > [Fri Feb 26 00:03:18.801825 2016] [so:debug] [pid 441] mod_so.c(266): > AH01575: loaded module authz_groupfile_module from > /usr/lib/apache2/modules/mod_authz_groupfile.so > [Fri Feb 26 00:03:18.852520 2016] [so:debug] [pid 441] mod_so.c(266): > AH01575: loaded module authz_user_module from > /usr/lib/apache2/modules/mod_authz_user.so > [Fri Feb 26 00:03:18.909582 2016] [so:debug] [pid 441] mod_so.c(266): > AH01575: loaded module authz_core_module from > /usr/lib/apache2/modules/mod_authz_core.so > [Fri Feb 26 00:03:18.964388 2016] [so:debug] [pid 441] mod_so.c(266): > AH01575: loaded module access_compat_module from > /usr/lib/apache2/modules/mod_access_compat.so > [Fri Feb 26 00:03:19.022456 2016] [so:debug] [pid 441] mod_so.c(266): > AH01575: loaded module auth_basic_module from > /usr/lib/apache2/modules/mod_auth_basic.so > [Fri Feb 26 00:03:19.081787 2016] [so:debug] [pid 441] mod_so.c(266): > AH01575: loaded module socache_shmcb_module from > /usr/lib/apache2/modules/mod_socache_shmcb.so > [Fri Feb 26 00:03:19.142737 2016] [so:debug] [pid 441] mod_so.c(266): > AH01575: loaded module reqtimeout_module from > /usr/lib/apache2/modules/mod_reqtimeout.so > [Fri Feb 26 00:03:19.205120 2016] [so:debug] [pid 441] mod_so.c(266): > AH01575: loaded module filter_module from > /usr/lib/apache2/modules/mod_filter.so > [Fri Feb 26 00:03:19.288557 2016] [so:debug] [pid 441] mod_so.c(266): > AH01575: loaded module deflate_module from > /usr/lib/apache2/modules/mod_deflate.so > [Fri Feb 26 00:03:19.355114 2016] [so:debug] [pid 441] mod_so.c(266): > AH01575: loaded module mime_module from /usr/lib/apache2/modules/mod_mime.so > [Fri Feb 26 00:03:19.428930 2016] [so:debug] [pid 441] mod_so.c(266): > AH01575: loaded module log_config_module from > /usr/lib/apache2/modules/mod_log_config.so > [Fri Feb 26 00:03:19.495578 2016] [so:debug] [pid 441] mod_so.c(266): > AH01575: loaded module env_module from /usr/lib/apache2/modules/mod_env.so > [Fri Feb 26 00:03:19.569779 2016] [so:debug] [pid 441] mod_so.c(266): > AH01575: loaded module headers_module from > /usr/lib/apache2/modules/mod_headers.so > [Fri Feb 26 00:03:19.641764 2016] [so:debug] [pid 441] mod_so.c(266): > AH01575: loaded module setenvif_module from > /usr/lib/apache2/modules/mod_setenvif.so > [Fri Feb 26 00:03:19.716852 2016] [so:debug] [pid 441] mod_so.c(266): > AH01575: loaded module version_module from > /usr/lib/apache2/modules/mod_version.so > [Fri Feb 26 00:03:20.213946 2016] [so:debug] [pid 441] mod_so.c(266): > AH01575: loaded module nss_module from /usr/lib/apache2/modules/libmodnss.so > [Fri Feb 26 00:03:20.307607 2016] [so:debug] [pid 441] mod_so.c(266): > AH01575: loaded module mpm_prefork_module from > /usr/lib/apache2/modules/mod_mpm_prefork.so > [Fri Feb 26 00:03:20.393425 2016] [so:debug] [pid 441] mod_so.c(266): > AH01575: loaded module unixd_module from /usr/lib/apache2/modules/mod_unixd.so > [Fri Feb 26 00:03:20.483090 2016] [so:debug] [pid 441] mod_so.c(266): > AH01575: loaded module status_module from > /usr/lib/apache2/modules/mod_status.so > [Fri Feb 26 00:03:20.578632 2016] [so:debug] [pid 441] mod_so.c(266): > AH01575: loaded module autoindex_module from > /usr/lib/apache2/modules/mod_autoindex.so > [Fri Feb 26 00:03:20.671102 2016] [so:debug] [pid 441] mod_so.c(266): > AH01575: loaded module dir_module from /usr/lib/apache2/modules/mod_dir.so > [Fri Feb 26 00:03:20.763080 2016] [so:debug] [pid 441] mod_so.c(266): > AH01575: loaded module alias_module from /usr/lib/apache2/modules/mod_alias.so > > Program received signal SIGSEGV, Segmentation fault. > 0x76b366dc in ?? () from /usr/lib/libssl3.so
Can I ask why you have NSSEnforceValidCerts off set? Is it because of an error that was logged, preventing the server from starting? rob _______________________________________________ Mod_nss-list mailing list [email protected] https://www.redhat.com/mailman/listinfo/mod_nss-list
