Re: [PATCH 60223] non-empty OpenSSL error queue preventing non-blocking reads through mod_ssl
On 11/11/2016 11:20 AM, Jacob Champion wrote: I suspect that these failures have nothing to do with this patch, but I just want to confirm that before it goes into trunk. Yeah, the failures show up with or without the patch, so I added a couple comments and committed. The SSL failures in the trunk test suite are going to make this difficult to test fully; has anyone seen them before or already have a fix tucked away privately? --Jacob
Re: [PATCH 60223] non-empty OpenSSL error queue preventing non-blocking reads through mod_ssl
On 11/11/2016 10:15 AM, William A Rowe Jr wrote: The patch looks reasonable to me. As you point out, fixing this in APR might be worthwhile, but doesn't protect mod_ssl from other OpenSSL consuming logic such as OpenLDAP auth, database connections or other plug-ins that may leave a lingering SSL_error state lying about. Looks good to me too -- thanks for the bump, Paul. I haven't committed this yet because of some SSL-related test failures in our suite when running against httpd-trunk: - mod_session_crypto appears to be segfaulting the server for some reason - mod_proxy tests hang the suite when run in -ssl test mode, and httpd complains about malformed request lines I suspect that these failures have nothing to do with this patch, but I just want to confirm that before it goes into trunk. --Jacob
Re: [PATCH 60223] non-empty OpenSSL error queue preventing non-blocking reads through mod_ssl
On Fri, Nov 11, 2016 at 9:01 AM, Paul Spangler wrote: > On 10/17/2016 2:04 PM, Paul Spangler wrote: > >> Hello, >> >> Due to the way OpenSSL stores errors in a per-thread queue, functions >> such as SSL_read followed by SSL_get_error may not produce the desired >> result if the error queue is not empty prior to calling SSL_read[1]. For >> example, a non-blocking read reports that no data is available by >> setting up SSL_get_error to return SSL_ERROR_WANT_READ. But if an error >> is already present in the queue, SSL_get_error sees that error instead >> and returns SSL_ERROR_SSL. >> >> I found at least one case where the error queue may be non-empty prior >> to a non-blocking read[2] that involves combining mod_session_crypto >> (which leaves the error queue non-empty) and the third-party >> mod_websocket (which uses non-blocking reads), resulting in the >> connection being closed. I included a potential patch to mod_ssl for >> consideration on the bug report that simply clears the error queue prior >> to any of the three SSL_* calls that mod_ssl makes. An ideal fix might >> be to keep the error queue empty at all times (i.e. patch the APR crypto >> library), but I propose that this patch is more robust in a modular >> environment. >> >> Thanks for your consideration. >> >> [1] >> https://www.openssl.org/docs/manmaster/ssl/SSL_get_error.html#DESCRIPTION >> [2] https://bz.apache.org/bugzilla/show_bug.cgi?id=60223 >> >> Anyone have any thoughts on this small patch? It addresses an issue with > OpenSSL's per-thread state causing connections to fail. > > > Regards, > Paul Spangler > LabVIEW R&D > National Instruments > The patch looks reasonable to me. As you point out, fixing this in APR might be worthwhile, but doesn't protect mod_ssl from other OpenSSL consuming logic such as OpenLDAP auth, database connections or other plug-ins that may leave a lingering SSL_error state lying about.
Re: mod_ftp segaults on rheloids
On Fri, Nov 11, 2016 at 2:24 AM, Benjamin Lefoul wrote: > Is this list still active? > Maybe it was not the right place to ask about mod_ftp? > It is the place to ask, and was on my list to respond to your question... sorry I'm overwhelmed with other code this week. The last 'release' is quite dated, and several things changed within httpd 2.4 that prove incompatible with the last official mod_ftp 0.9.6 release. I did try to push out another release, but it never got enough votes to be 'released' by the community, and since has undergone a number of bug fixes to work with the current httpd 2.4.23 (although it should still be compatible with an older httpd release like 2.4.6.) http://svn.apache.org/repos/asf/httpd/mod_ftp/trunk/ is the repository of the current source code. The primary defect I'm aware of in an incompatibility with mod_ftp and the httpd core that prevents us from successfully serving FTP Explicit SSL connections. An implicit SSL connection, e.g. ftps on port 990 (using TLS from the time of connecting) is functional. Once other httpd projects are wrapped up, I'm coming back to httpd 2.4 and mod_ftp to figure out how to get Explicit SSL functioning again. If you would grab the latest development code I'd love to read about your success or any defects you encounter. Cheers, Bill
Re: [PATCH 60223] non-empty OpenSSL error queue preventing non-blocking reads through mod_ssl
On 10/17/2016 2:04 PM, Paul Spangler wrote: Hello, Due to the way OpenSSL stores errors in a per-thread queue, functions such as SSL_read followed by SSL_get_error may not produce the desired result if the error queue is not empty prior to calling SSL_read[1]. For example, a non-blocking read reports that no data is available by setting up SSL_get_error to return SSL_ERROR_WANT_READ. But if an error is already present in the queue, SSL_get_error sees that error instead and returns SSL_ERROR_SSL. I found at least one case where the error queue may be non-empty prior to a non-blocking read[2] that involves combining mod_session_crypto (which leaves the error queue non-empty) and the third-party mod_websocket (which uses non-blocking reads), resulting in the connection being closed. I included a potential patch to mod_ssl for consideration on the bug report that simply clears the error queue prior to any of the three SSL_* calls that mod_ssl makes. An ideal fix might be to keep the error queue empty at all times (i.e. patch the APR crypto library), but I propose that this patch is more robust in a modular environment. Thanks for your consideration. [1] https://www.openssl.org/docs/manmaster/ssl/SSL_get_error.html#DESCRIPTION [2] https://bz.apache.org/bugzilla/show_bug.cgi?id=60223 Anyone have any thoughts on this small patch? It addresses an issue with OpenSSL's per-thread state causing connections to fail. Regards, Paul Spangler LabVIEW R&D National Instruments
Re: mod_ftp segaults on rheloids
On Fri, 2016-11-11 at 13:22 +0100, Rainer Canavan wrote: > On Fri, Nov 11, 2016 at 10:27 AM, Benjamin Lefoul > wrote: > > > > On Fri, 2016-11-11 at 08:53 +, Nick Kew wrote: > > > > > > Did you build mod_ftp yourself on a distro-provided httpd? > > Yes. I did build myself from the latest svn. > > I installed: > > httpd-devel- 2.4.6-40.sl7.4.x86_64 from Scientific Linux 7 > > httpd-devel-2.4.6- 40.el7.centos.4.x86_64 on centOS7 > > (so, the same, essentially), as well as the "Development Tools" > > group. > > If you have a core dump, try analyzing it with gdb, otherwise, since > you can apparently trivially reproduce the segfault, just attach gdb > to > any httpd process (except maybe the one that has pid 1 as its parent) > with > > gdb $(which httpd) > > it will tell you that lots of debug symbols are absent, and how to > run > debuginfo-install > to install them. Do that as instructed, re-run gdb, verify that no > "symbols missing" messages > are printed on startup. Execute "cont" at the gdb prompt and start > causing segfaults again > until gdb returns to the prompt. Do a "bt full". Not sure if httpd- > dev > is the right place, it > might be a bug in CentOS and/or the ancient httpd they're using. > Thanks for your help and patience. I wasn't able to get the prompt back so after a while I just C-c'd. Any comment? This is what I get: (gdb) cont Continuing. Detaching after fork from child process 13499. Detaching after fork from child process 13500. Detaching after fork from child process 13501. Detaching after fork from child process 13502. Detaching after fork from child process 13503. ^C Program received signal SIGINT, Interrupt. 0x7f598fc773f3 in __select_nocancel () at ../sysdeps/unix/syscall- template.S:81 81 T_PSEUDO (SYSCALL_SYMBOL, SYSCALL_NAME, SYSCALL_NARGS) (gdb) bt full #0 0x7f598fc773f3 in __select_nocancel () at ../sysdeps/unix/syscall-template.S:81 No locals. #1 0x7f599038f585 in apr_sleep (t=t@entry=100) at time/unix/time.c:246 tv = {tv_sec = 0, tv_usec = 713807} #2 0x7f59916b1dd1 in ap_wait_or_timeout (status=status@entry=0x7ff dd22c85f8, exitcode=exitcode@entry=0x7ffdd22c85fc, ret=ret@entry=0x7ffd d22c8600, p=, s=) at mpm_common.c:195 rv = #3 0x7f5986e8a13e in prefork_run (_pconf=, plog=, s=) at prefork.c:1016 status = 11 pid = {pid = -1, in = 0x7f599308ef58, out = 0x34a934a9, err = 0x7f599300c340} child_slot = exitwhy = APR_PROC_SIGNAL processed_status = index = remaining_children_to_start = 0 rv = #4 0x7f59916b15be in ap_run_mpm (pconf=pconf@entry=0x7f5992fe3158, plog=0x7f5993010378, s=0x7f599300c340) at mpm_common.c:96 pHook = n = 0 rv = -1 #5 0x7f59916aab46 in main (argc=2, argv=0x7ffdd22c88f8) at main.c:777 c = 68 'D' showcompile = 0 showdirectives = 0 confname = 0x7f59916e68af "conf/httpd.conf" def_server_root = 0x7f59916e68a4 "/etc/httpd" temp_error_log = 0x0 error = process = 0x7f5992fe1238 pconf = 0x7f5992fe3158 plog = 0x7f5993010378 ptemp = 0x7f599300e368 pcommands = 0x7f5993005268 opt = 0x7f5993005358 rv = mod = 0x7f5991904098 opt_arg = 0x7ffdd22c9f7d "FOREGROUND" signal_server = And the error log: [Fri Nov 11 12:50:45.402984 2016] [mpm_prefork:notice] [pid 13481] AH00163: Apache/2.4.6 (CentOS) mod_ftp/1.0.1-dev OpenSSL/1.0.1e-fips mod_fcgid/2.3.9 configured -- resuming normal operations [Fri Nov 11 12:50:45.403003 2016] [core:notice] [pid 13481] AH00094: Command line: '/usr/sbin/httpd -D FOREGROUND' [Fri Nov 11 12:50:45.427847 2016] [ftp:notice] [pid 13482] FTP low numbered port daemon waiting for port requests [Fri Nov 11 12:51:26.868223 2016] [core:notice] [pid 13481] AH00052: child pid 13484 exit signal Segmentation fault (11) [Fri Nov 11 12:51:56.902013 2016] [core:notice] [pid 13481] AH00052: child pid 13485 exit signal Segmentation fault (11) [Fri Nov 11 12:52:15.923595 2016] [core:notice] [pid 13481] AH00052: child pid 13486 exit signal Segmentation fault (11) [Fri Nov 11 12:52:22.932293 2016] [core:notice] [pid 13481] AH00052: child pid 13487 exit signal Segmentation fault (11) [Fri Nov 11 12:52:25.935810 2016] [core:notice] [pid 13481] AH00052: child pid 13488 exit signal Segmentation fault (11)
Re: mod_ftp segaults on rheloids
On Fri, 2016-11-11 at 08:53 +, Nick Kew wrote: > On Fri, 2016-11-11 at 08:24 +, Benjamin Lefoul wrote: > > > > Is this list still active? > > Maybe it was not the right place to ask about mod_ftp? > > I guess noone picked up the baton. Too many blanks to fill. > > Did you build mod_ftp yourself on a distro-provided httpd? Yes. I did build myself from the latest svn. I installed: httpd-devel-2.4.6-40.sl7.4.x86_64 from Scientific Linux 7 httpd-devel-2.4.6-40.el7.centos.4.x86_64 on centOS7 (so, the same, essentially), as well as the "Development Tools" group. > What toolchain did you use for that? > What happens if you build everything yourself, I'd rather focus on trying to get it to work on the distro provided apache for this week (I already have some services neatly set there). (I'll append here if I can clear some time to try later this month.) > or alternatively use everything from the distro? > Where's the traceback from your segfault? > Can you help me with this? I guess I should run gdb and attach the correct process, but how do I set the debug flags? Thank you very much, Benjamin
Re: mod_ftp segaults on rheloids
On Fri, 2016-11-11 at 08:24 +, Benjamin Lefoul wrote: > Is this list still active? > Maybe it was not the right place to ask about mod_ftp? I guess noone picked up the baton. Too many blanks to fill. Did you build mod_ftp yourself on a distro-provided httpd? What toolchain did you use for that? What happens if you build everything yourself, or alternatively use everything from the distro? Where's the traceback from your segfault? -- Nick Kew
Re: mod_ftp segaults on rheloids
Is this list still active? Maybe it was not the right place to ask about mod_ftp? Benjamin On Tue, 2016-11-08 at 09:36 +0100, Benjamin Lefoul wrote: > Hi, > > I have built mod_ftp on fresh RHELoids (tried on both CentOS7 and > Scientific Linux 7). > Whether I try the default conf or my own, the module always > immediately segfaults: > > [Tue Nov 08 08:15:52.798201 2016] [suexec:notice] [pid 1225] AH01232: > suEXEC mechanism enabled (wrapper: /usr/sbin/suexec) > [Tue Nov 08 08:15:52.890162 2016] [auth_digest:notice] [pid 1225] > AH01757: generating secret for digest authentication ... > [Tue Nov 08 08:15:52.891402 2016] [lbmethod_heartbeat:notice] [pid > 1225] AH02282: No slotmem from mod_heartmonitor > [Tue Nov 08 08:15:52.898709 2016] [ftp:notice] [pid 1333] FTP low > numbered port daemon waiting for port requests > [Tue Nov 08 08:15:52.914261 2016] [mpm_prefork:notice] [pid 1225] > AH00163: Apache/2.4.6 (CentOS) mod_ftp/1.0.1-dev OpenSSL/1.0.1e-fips > mod_fcgid/2.3.9 configured -- resuming normal operations > [Tue Nov 08 08:15:52.914279 2016] [core:notice] [pid 1225] AH00094: > Command line: '/usr/sbin/httpd -D FOREGROUND' > [Tue Nov 08 08:25:02.543360 2016] [core:notice] [pid 1225] AH00052: > child pid 1337 exit signal Segmentation fault (11) > [Tue Nov 08 08:25:32.577127 2016] [core:notice] [pid 1225] AH00052: > child pid 1338 exit signal Segmentation fault (11) > [Tue Nov 08 08:25:40.586853 2016] [core:notice] [pid 1225] AH00052: > child pid 1339 exit signal Segmentation fault (11) > > Each of these segfaults is me typing "ls" in lftp. > I juste want to get the ftp feature running. I can provide more info > if you tell me how :) > > Thanks! > > Benjamin Lefoul > nWISE AB