https://bz.apache.org/bugzilla/show_bug.cgi?id=66168
--- Comment #3 from [email protected] --- Created attachment 40249 --> https://bz.apache.org/bugzilla/attachment.cgi?id=40249&action=edit Minimal FastCGI responder sending an early empty FCGI_STDOUT record (reproducer) Root cause identified and reproduced without PHP on httpd 2.4.70. Environment ----------- httpd 2.4.70, mpm_event, mod_http2, mod_proxy_fcgi -> PHP-FPM mod_brotli enabled via AddOutputFilterByType (mod_filter): AddOutputFilterByType BROTLI_COMPRESS text/html text/plain ... libbrotli 1.1.0-2ubuntu0.1, Ubuntu 24.04.5 LTS PHP-FPM 7.4.33 as real-world trigger (see below) Summary ------- The segfault is a NULL dereference of the brotli encoder state. It happens when mod_brotli receives data after it has already processed EOS: 1. The FastCGI backend sends an empty FCGI_STDOUT record in the middle of the response and continues with further FCGI_STDOUT records afterwards. PHP-FPM does exactly this due to PHP bug #72185 (all 7.x, 8.0 < 8.0.20, 8.1 < 8.1.7): when the stream switches between STDOUT and STDERR while its 8 KB output buffer is full, it emits an empty record. 2. mod_proxy_fcgi treats the empty FCGI_STDOUT record as end of stream and passes EOS down the output filter chain, but keeps reading from the backend and forwards the following FCGI_STDOUT records as well, i.e. data after EOS. 3. mod_brotli finishes and destroys the encoder on EOS (ctx->state = NULL) but stays in the filter chain (wrapped by mod_filter's filter_harness). The next data bucket reaches process_chunk(), and BrotliEncoderCompressStream(NULL, BROTLI_OPERATION_PROCESS, ...) dereferences NULL. The same call chain was already reported in bug 63216 (2.4.38) and in this bug (2.4.41 - 2.4.57). Reproduction without PHP ------------------------ The attached fcgi_early_eos.py (Python 3, stdlib only) is a minimal FastCGI responder. With mode=early it sends a header record, 20 FCGI_STDOUT records, an empty FCGI_STDOUT record, 5 more FCGI_STDOUT records and a final "END" line. python3 fcgi_early_eos.py --listen 127.0.0.1:9123 (as non-root) ProxyPass "/fcgitest" "fcgi://127.0.0.1:9123/" curl --http1.1 -o /dev/null -H 'Accept-Encoding: br' 'https://<host>/fcgitest?mode=early' curl --http2 -o /dev/null -H 'Accept-Encoding: br' 'https://<host>/fcgitest?mode=early' Result: both requests crash the child process (AH00052). This happens with HTTP/1.1 as well as HTTP/2, so mod_http2 is not involved. Controls: - mode=normal (no early empty record), brotli: no crash - mode=early, Accept-Encoding: gzip (mod_deflate): no crash - mode=early, Accept-Encoding: identity: response truncated at the early EOS (final "END" line missing) Reproduction with PHP-FPM ------------------------- PHP-FPM <7.x.y>, log_errors on (messages are sent as FCGI_STDERR): <?php ini_set('display_errors', 1); for($i=0;$i<500;$i++) { f; } ?> Whether the empty record is hit depends on the exact byte layout of the output (script path length, line numbers, headers). With a different path the loop count may need to be varied (e.g. 100..1000). This explains why the crash appears sporadic and why it looked DocumentRoot-dependent in the original report. Analysis details ---------------- Crash on 2.4.70 (no debug symbols for httpd; the symbolized backtrace in the original report shows the same call chain): #0 BrotliEncoderCompressStream+45: mov 0x1b3c(%rdi),%r8d rdi = 0 (encoder state), rsi = 0 (BROTLI_OPERATION_PROCESS) #1 mod_brotli, process_chunk (avail_out = 0, next_out = NULL) #2 mod_filter, filter_harness #3 mod_proxy_fcgi, dispatch brotli_ctx_t in frame #1: state = NULL bb = valid pointer total_in = 15957 (= 81 x 197 bytes) total_out = 1043 The encoder had already produced output and was cleaned up regularly before the crash; this is not a corrupted context. FilterTrace BYTYPE:BROTLI_COMPRESS 1 on the crashing request: (TRANSIENT 197, FLUSH) x 81, EOS, TRANSIENT 197 -> segfault strace of the PHP-FPM worker: two write()s of 8200 bytes (the full 8192 byte buffer plus an appended record header without content), ending in 01 07 00 01 00 00 00 00 (empty FCGI_STDERR) 01 06 00 01 00 00 00 00 (empty FCGI_STDOUT) followed by further FCGI_STDOUT records. Impact ------ Only the child process crashes (AH00052 is logged by the parent), but all other connections handled by that child are lost. On shared hosting any customer running an affected PHP version can trigger this with ordinary PHP warnings. Independent of compression, clients receive a truncated response at the early EOS. Suggested fixes --------------- - mod_proxy_fcgi: after an empty FCGI_STDOUT record, do not pass further STDOUT data down the filter chain; discard it and log once that the backend sent data after the end of the stream. - mod_brotli: never call the encoder with a NULL state; ignore buckets that arrive after EOS instead of crashing. Workarounds ----------- - SetEnv no-brotli 1 for responses from affected backends - keep PHP messages off FCGI_STDERR (error_log to a file or fastcgi.logging = 0), which avoids the stream switches that trigger the empty record in PHP-FPM - upgrade PHP-FPM to 8.0.20 / 8.1.7 or later -- You are receiving this mail because: You are the assignee for the bug. --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
