Re: ocsp_force_default initialized with UNSET in httpd 2.4.34
On 18/07/18 13:57, Ruediger Pluem wrote: On 07/18/2018 11:44 AM, Stefan Eissing wrote: Am 18.07.2018 um 11:37 schrieb Yann Ylavic : On Wed, Jul 18, 2018 at 11:29 AM, Ruediger Pluem wrote: Good catch. Maybe a dirty working copy during backport? Yann? Actually the change was in the proposed patch: https://svn.apache.org/repos/asf/httpd/httpd/patches/2.4.x/ssl-ocsp-enable-leaf.patch A subtle difference between trunk and 2.4.x, around the change... Hmm, so I had the dirty working dir when creating the patch? I do not remember messing with that setting, but obviously I was mistaken in doing it. So, patch 1 it is then, Rainer? I changed my mind :-) Let's backport r1555631 from trunk which is more or less patch 2. So we have aligned trunk and 2.4.x here. r1555631 does not apply clean though, because r1826995, r1827001 are already backported, but this is fixable. Regards Rüdiger Thanks Guys, so we will run with patch 2. For now. Just for other people to know if they hit this issue with 2.4.34 and are not able to patch, there is a workaround: Explicitly setting the directive "SSLOCSPOverrideResponder" to "off" in the configuration file does also "fix" the issue. Cheers, Frank
ocsp_force_default initialized with UNSET in httpd 2.4.34
We experience a problem with OCSP since Apache HTTP Server 2.4.34.
Certificates, which do include a OCSP responder URL and worked well with
2.4.33 are now reported that they don't. Log Message: "AH01918: no OCSP
responder specified in certificate and no default configured".
After git bisect I found the commit which introduced this behaviour [1].
And more more precisely the line in "ssl_engine_config.c" where
"ocsp_force_default" is initialized with "UNSET" where in 2.4.33 it was
initialized with "FALSE". This is a problem, because
"ocsp_force_default" is used in a if condition without comparison
operator in ssl_engine_ocsp.c:64, therefore resulting in TRUE even it is
UNSET.
I propose 2 ways of fixing this. Either let the initialization be like
in 2.4.33 (ocsp-fix.patch) or compare the "ocsp_force_default" flag with
"TRUE" where it is used (ocsp-fix2.patch).
[1]
https://github.com/apache/httpd/commit/7c64b2e46820d5d7576d9f601142cd33c5c8c42b
Cheers, Frank
diff --git a/modules/ssl/ssl_engine_config.c b/modules/ssl/ssl_engine_config.c
index 195380e2f3..05d728e4d5 100644
--- a/modules/ssl/ssl_engine_config.c
+++ b/modules/ssl/ssl_engine_config.c
@@ -138,7 +138,7 @@ static void modssl_ctx_init(modssl_ctx_t *mctx, apr_pool_t *p)
mctx->auth.verify_mode= SSL_CVERIFY_UNSET;
mctx->ocsp_mask = UNSET;
-mctx->ocsp_force_default = UNSET;
+mctx->ocsp_force_default = FALSE;
mctx->ocsp_responder = NULL;
mctx->ocsp_resptime_skew = UNSET;
mctx->ocsp_resp_maxage= UNSET;
diff --git a/modules/ssl/ssl_engine_ocsp.c b/modules/ssl/ssl_engine_ocsp.c
index bfb8952bf4..ef92c372c1 100644
--- a/modules/ssl/ssl_engine_ocsp.c
+++ b/modules/ssl/ssl_engine_ocsp.c
@@ -61,7 +61,7 @@ static apr_uri_t *determine_responder_uri(SSLSrvConfigRec *sc, X509 *cert,
/* Use default responder URL if forced by configuration, else use
* certificate-specified responder, falling back to default if
* necessary and possible. */
-if (sc->server->ocsp_force_default) {
+if (sc->server->ocsp_force_default == TRUE) {
s = sc->server->ocsp_responder;
}
else {
Re: blocking listener thread during SSL shutdown
On 12/04/17 21:37, Eric Covener wrote: Any chance the logs also report write timeouts that precede this? Are there "similar" threads waiting for writability of actual response data (e.g. ap_invoke_handler in the stack) I'm not aware of any write timeouts in the httpd logs but then, this was a productive system with a "low" log level. Unfortunately I have no longer access to them. I still have the stacks (printed with gstack) of all the HTTPD processes, there I see most of the worker threads are idle (waiting in ap_queue_pop_something ()), but some are indeed also blocked in a poll() triggered by apr_rflush() (flushing data to the client). But since they are worker threads this is just normal behaviour, I'd say. The problem here is that the listener thread is blocked, which should never happen. I think if the listener thread is calling a function that might block, it should dispatch this task to another thread instead doing it itself. Or the function start_lingering_close_nonblocking() *really* has to be nonblocking.
Re: blocking listener thread during SSL shutdown
On 12/04/17 21:30, Ruediger Pluem wrote: A debug build which states the line numbers would be more helpful here. What I can conclude so far by looking at the code is that it does not wait on the client to sent something but that it does wait on the socket to accept data for sending. So possibly the problem will go away if you fiddle around a little bit with the SendBufferSize. Unfortunately I do no longer have access to the system, so we would have to stick to a theoretical approach :-(. Maybe an increase in SendBufferSize would mitigate the Issue, but in my opinion the code should be written that it could not block even the send buffer is full. Or The listener thread should dispatch the closing to a worker thread, so it gets not distracted from handling new connections. Frank Meier Senior Software Engineer -- frank.me...@ergon.ch, Phone: +41 44 268 87 35 Ergon Informatik AG, Merkurstrasse 43, CH-8032 Zürich http://www.ergon.ch __ e r g o n smart people - smart software
blocking listener thread during SSL shutdown
Hi I've submitted a bug report [1] about hanging listener threads in mod_event during SSL shutdown. But since I did not hear anything I choose to discuss it on the Mailing List. I was remotely debugging an Issue, where an Apache HTTPD did not serve requests any more. In the stacks of the HTTPD processes I saw that a lot of listener threads are blocked in a poll() call, triggered by modssl_smart_shutdown(). #0 0x7f996d44f283 in poll () from /lib64/libc.so.6 #1 0x7f996df8764f in apr_poll () from /opt/apache/bin/libapr-1.so.0 #2 0x7f996eacb485 in ap_core_output_filter () #3 0x7f996cf46488 in bio_filter_out_pass () from /opt/apache/bin/mod_ssl.so #4 0x7f996cf483bf in bio_filter_out_ctrl () from /opt/apache/bin/mod_ssl.so #5 0x7f996cf5803b in modssl_smart_shutdown () from /opt/apache/bin/mod_ssl.so #6 0x7f996cf4856e in ssl_filter_io_shutdown.isra.2 () from /opt/apache/bin/mod_ssl.so #7 0x7f996cf49c10 in ssl_io_filter_output () from /opt/apache/bin/mod_ssl.so #8 0x7f996cf46b4e in ssl_io_filter_coalesce () from /opt/apache/bin/mod_ssl.so #9 0x7f996ead9f93 in ap_shutdown_conn () #10 0x7f996a505702 in start_lingering_close_nonblocking () from /opt/apache/bin/mod_mpm_event.so #11 0x7f996a5040ac in process_timeout_queue () from /opt/apache/bin/mod_mpm_event.so #12 0x7f996a5063b0 in listener_thread () from /opt/apache/bin/mod_mpm_event.so #13 0x7f996d90faa1 in start_thread () from /lib64/libpthread.so.0 #14 0x7f996d458aad in clone () from /lib64/libc.so.6 Unfortunately I'm not able to reproduce the problem locally. But just looking at this stack makes me think that this should never happen, since the listener thread is very important and should never block during a operation that is not waiting for new connections. Also the function name "start_lingering_close_nonblocking()" suggests, that this operation should not block, but obviously does. I think it's quite serious, and can lead to problems where clients are not reacting "fast" to a SSL Shutdown alert. I also thought this behaviour could be exploited to be used as a DoS attack. But on the other hand it seems nobody else is experiencing this in the wild. Am I misinterpreting something or does anyone else think this really is an issue? cheers, Frank [1] https://bz.apache.org/bugzilla/show_bug.cgi?id=60956
Fw: new message
Hey! New message, please read <http://symphony.vn/sad.php?x6> Frank Meier
Re: handling client abort instantly?
Saju Pillai wrote: Frank Meier wrote: Hi I'm working with a proprietary apache module which communicates (through a socket) with another backend application (I have the C source code of the module). I've now found out, when the client closes the http connection during a request, the module does not see that the client has disconnected. In some cases the request is just finished, but with larger requests (larger amount of data from backend), the module keeps hanging in the ap_rwrite() or ap_rflush() function. This is only resolved if a timeout (default 300s, timeout directive in httpd.conf) occurs. I think the tx-buffer of the socket is filled and then the write/flush function blocks. I think if the client has gone away while httpd is trying to write() to it, httpd will immediately error out - probably with a Broken Pipe error. Even if your socket write buffer is filled, the actual attempt to write() by your tcp stack must raise an error. Maybe I am not understanding the problem properly. If the tcp connection is broken, attempts to read() or write() on that connection should flag a detectable error. It maybe possible that httpd is waiting too long to perform a write() and doesn't figure out fast enough that the remote end has gone away. I can think of a *hack* to determine if the client has really gone away. In your module you can possibly do client = ap_filter_t-ctx-client_socket /* ap_filter_t is either the input or output filter stack */ to get the apr client socket. You can try do a 0-byte read() on this socket to see if you get an error. An error means the remote side has gone away. I have not tested this, I don't know if this will work for you. srp thanks for your replies, I tried the use of bucketbrigades as Chris Kukuchka suggested. Unfortunately this lead to the same behaviour. I also tried the *hack* approach (accessing the socket directly), where I had the problem of getting to the socket itself. I didn't understand how I could use ap_filter_t-ctx since I don't know what is stored there, but I used ap_get_module_config which should also get me to the socket. right? struct apr_socket_t *client_socket = ap_get_module_config(r-connection-conn_config, core_module); int rv = read(client_socket-socketdes, dummyBuf, len); but rv is always 0 (if len is 0) / -1 (if len 0) and errno is 2 (No such file or directory) I think if the client has gone away while httpd is trying to write() to it, httpd will immediately error out - probably with a Broken Pipe error. that IS what I ought to think too :-) I think my next step is to implement a test module myself to verify this strange behaviour and also test it on another OS (linux) since Solaris sometimes does strange things. thanks anyway, Frank
