https://bz.apache.org/bugzilla/show_bug.cgi?id=66160
Bug ID: 66160
Summary: Apache HTTPD stops serving and freezes HTTP/2
connections
Product: Apache httpd-2
Version: 2.4.53
Hardware: PC
OS: Solaris
Status: NEW
Severity: normal
Priority: P2
Component: mod_http2
Assignee: [email protected]
Reporter: [email protected]
Target Milestone: ---
Created attachment 38334
--> https://bz.apache.org/bugzilla/attachment.cgi?id=38334&action=edit
Analysis data from Apache HTTP being in a state where a subset of the incomming
HTTP/2 connection freeze
We are running Apache 2.4.53 on Oracle Solaris 11.4 SRU 44 on x64 hardware.
Apache has been compiled by Oracle. The server mainly runs an e-learning
platform running Moodle, Nextcloud, Mahara, Guacamole and some more tools for
about 2000 extra-occupational mainly distant-learning students.
HTTP/2 is active.
After several weeks of having a perfectly running system, Apache HTTPD seems to
stop serving some of the newly incoming HTTP requests using HTTP/2. The
affected TCP connections stay frozen in the ESTABLISHED state on both, client
and server.
Apache HTTPD seems to stay in this "erratic state" (where some connections are
served fine and some are frozen) until the web server process is restarted by
issuing a "svcadm restart apache24". After restarting it, the system is running
fine for two or three weeks until the problem reoccurs out of nothing.
When forcing HTTP/1.1 connections, the problem cannot be triggered even when
Apache HTTPD is in the above-mentioned erratic state. It seems to be strongly
related to HTTP/2.
Our servers with a similar configuration with HTTP/2 being deactivated don't
seem to be affected - but they are in general loaded less intensively.
You'll find an attachment consisting on four parts:
1) "truss" output of all running Apache HTTPD processes when a HTTP/2
connection comes in which is successfully served.
2) "truss" output of all running Apache HTTPD processes when a HTTP/2
connection comes in which is frozen.
3) "pstack" analysis of the affected process and it's threads.
4) An output of Apache HTTPD's "server-status" when it was in the erratic
state.
When looking at c), it seems that the problem can be narrowed down to the calls
fired by thread #32 of process #8544:
------------ lwp# 32 / thread# 32 ---------------
00007fb3cfc73147 lwp_park (0, 7fb3c4e16a20, 0)
00007fb3cfc6b8d7 cond_wait_queue () + 67
00007fb3cfc6bd5a cond_wait_common () + 1ea
00007fb3cfc6bfdd __cond_timedwait () + 6d
00007fb3cfc6c08a cond_timedwait () + 2a
00007fb3cfc6c0c9 pthread_cond_timedwait () + 9
00007fb3cf6428bf apr_thread_cond_timedwait () + 6f
00007fb3c8c327db h2_mplx_m_out_trywait () + bb
00007fb3c8c40a49 h2_session_process () + 1269
00007fb3c8c28716 h2_conn_run () + 96
00007fb3c8c2fea2 h2_h2_process_conn () + 9b2
00000000004ac08b ap_run_process_connection () + 3b
00007fb3ca00adee process_socket () + 3fe
00007fb3ca00e365 worker_thread () + 355
0000000000470d89 thread_start () + 19
00007fb3cf65659e dummy_worker () + e
00007fb3cfc72de3 _thrp_setup () + b3
00007fb3cfc73100 _lwp_start ()
It seems to me that the "h2_mplx_m_out_trywait()" function, which is
implemented in "httpd-2.4/modules/http2/h2_mplx.c", gets into a deadlock when
it wants to enter the MUTEX section, possibly by issuing the "H2_MPLX_ENTER(m)"
command.
The problem is: I had to restart Apache HTTP on the affected machine because it
is a production server. I hope that I could gather relevant data while it was
in the strange state. If not, we unfortunately have to wait until it re-occurs.
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
