https://bz.apache.org/bugzilla/show_bug.cgi?id=69666
Bug ID: 69666
Summary: AH00144: couldn't grab the accept mutex
Product: Apache httpd-2
Version: 2.4.63
Hardware: PC
OS: Mac OS X 10.1
Status: NEW
Severity: normal
Priority: P2
Component: Core
Assignee: bugs@httpd.apache.org
Reporter: g...@cpanel.net
Target Milestone: ---
A few years ago, A recent APR upgrade (1.6 to 1.7) caused this to start
happening on many of our custs running different Linux distros (eg: CentOS6, 7)
[Thu May 30 08:21:07.287609 2019] [mpm_prefork:emerg] [pid 13019] (22)Invalid
argument: AH00144: couldn't grab the accept mutex
[Thu May 30 08:21:07.289188 2019] [mpm_prefork:emerg] [pid 13016] (22)Invalid
argument: AH00144: couldn't grab the accept mutex
[Thu May 30 08:21:07.344749 2019] [core:alert] [pid 13014] AH00050: Child 13016
returned a Fatal error... Apache is exiting!
The APR change in question was likely this commit:
https://github.com/apache/apr/commit/9152473c94015f98c7976f28b45a4fb6be8a4e39
When we rebuilt Apache against APR 1.7, httpd -V was showing this:
-D APR_USE_PROC_PTHREAD_SERIALIZE
APR 1.6.x had this instead:
-D APR_USE_SYSVSEM_SERIALIZE
The 'fix' was for us to force SYSVSEM as the default locking mech on APR 1.7,
and moreso recently, we decided to try this again w/ Apache 2.4.63 and the
issue reappeared.
This was occuring with PreFork, Worker, and even Event.
The only thing I've been able to see on my sandbox, so far, is using this
bpftrace one liner to hook the apr calls and look at the return code;
bpftrace -e '
> uprobe:/opt/cpanel/ea-apr16/lib64/libapr-1.so.0:apr_global_mutex_lock,
> uprobe:/opt/cpanel/ea-apr16/lib64/libapr-1.so.0:apr_global_mutex_unlock
> /comm == "httpd"/
> {
> @calls[tid] = nsecs;
> }
>
> uretprobe:/opt/cpanel/ea-apr16/lib64/libapr-1.so.0:apr_global_mutex_lock,
> uretprobe:/opt/cpanel/ea-apr16/lib64/libapr-1.so.0:apr_global_mutex_unlock
> /comm == "httpd"/
> {
> if (retval != 0) {
> printf("mutex ERROR! PID %d TID %d ret=%d func=%s\n", pid, tid,
> retval, probe);
> }
> delete(@calls[tid]);
> }
> '
On my sandbox, I randomly see this (using ab) and sending apache a HUP.. It
happens maybe 2x a day, but there's no "couldn't grab the accept mutex" error
in Apache's error_log at all like it was reported.
mutex ERROR! PID 3440229 TID 3440229 ret=43
func=uretprobe:/opt/cpanel/ea-apr16/lib64/libapr-1.so.0:apr_global_mutex_lock
mutex ERROR! PID 3440229 TID 3440229 ret=22
func=uretprobe:/opt/cpanel/ea-apr16/lib64/libapr-1.so.0:apr_global_mutex_unlock
We're not using any 3rd party modules or anything that would cause this
behavior, once again, I can't reproduce it locally but I have seen it on dozens
of servers.
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscr...@httpd.apache.org
For additional commands, e-mail: bugs-h...@httpd.apache.org
