https://bz.apache.org/bugzilla/show_bug.cgi?id=69668
Bug ID: 69668
Summary: Data race between listening thread and exiting thread
in one-process mode
Product: Apache httpd-2
Version: 2.5-HEAD
Hardware: PC
OS: Linux
Status: NEW
Severity: minor
Priority: P2
Component: Core
Assignee: [email protected]
Reporter: [email protected]
Target Milestone: ---
When Apache web server (httpd) is run in the "one-process" mode (for debugging,
the -X command-line option), then it can experience a data race on exit.
The bug is a data race:
* listening thread starts accepting a new client connection, and
* some other thread performs exit() to terminate the whole process.
At this point, the other thread calls ap_terminate() because it was registered
via atexit() glibc function. ap_terminate() frees all memory pools. At the same
time, the listening thread tries to use data from these memory pools, in
particular, the data of queue_info->recycled_pools. Since the memory was
already freed by the other thread, the listening thread's memory pool access
results in a heap use-after-free.
I consider it a benign bug because (a) it only triggers in the “one-process”
mode that is used only for debugging, (b) it happens only on process
termination, (c) it requires another thread to call exit(), (d) it happens
rarely when the listening thread wakes up to accept a client connection. In
particular, item c is a strong requirement: in my experiments, I LD_PRELOAD a
preeny library that calls exit() on a particular socket shutdown; see
https://github.com/zardus/preeny/blob/master/src/desock.c
I am not aware of other, more realistic scenarios when this bug triggers. But
since it happened in my debugging case, and it looks like a legit (though
probably latent) bug, I decided to post this report.
This was detected on Amazon Linux 2023 system, on httpd commit
e36237899d56fb457f43735b75e085ec3c320ecc.
---
This bug was found when httpd was built for debugging purposes, using the
following config:
./configure --with-included-apr --enable-pool-debug
I followed the INSTALL instructions. Latest APR version was used by performing:
git clone https://github.com/apache/apr.git srclib/apr
Here is a simplified AddressSanitizer trace that fully explains the bug:
ERROR: AddressSanitizer: heap-use-after-free on address xyz
READ of size 8 at thread T71
#0 ap_queue_info_pop_pool /httpd/server/mpm_fdqueue.c:275
#1 listener_thread /httpd/server/mpm/event/event.c:2228
#2 dummy_worker /httpd/srclib/apr/threadproc/unix/thread.c:165
freed by thread T46 here:
#1 pool_clear_debug /httpd/srclib/apr/memory/unix/apr_pools.c:1906
#2 pool_destroy_debug /httpd/srclib/apr/memory/unix/apr_pools.c:1965
#3 pool_clear_debug /httpd/srclib/apr/memory/unix/apr_pools.c:1880
#4 pool_destroy_debug /httpd/srclib/apr/memory/unix/apr_pools.c:1965
#5 pool_clear_debug /httpd/srclib/apr/memory/unix/apr_pools.c:1880
#6 pool_destroy_debug /httpd/srclib/apr/memory/unix/apr_pools.c:1965
#7 apr_pool_destroy_debug /httpd/srclib/apr/memory/unix/apr_pools.c:2014
#8 ap_terminate /httpd/os/unix/unixd.c:522
#9 __run_exit_handlers (/lib64/libc.so.6+0x5750c)
previously allocated by thread T71 here:
#0 malloc
#1 pool_alloc /httpd/srclib/apr/memory/unix/apr_pools.c:1787
#2 ap_queue_info_push_pool /httpd/server/mpm_fdqueue.c:237
#3 listener_thread /httpd/server/mpm/event/event.c:2283
#4 dummy_worker /httpd/srclib/apr/threadproc/unix/thread.c:165
Thread T71 created by T45 here:
#1 apr_thread_create /httpd/srclib/apr/threadproc/unix/thread.c:235
#2 create_listener_thread /httpd/server/mpm/event/event.c:2608
#3 start_threads /httpd/server/mpm/event/event.c:2845
#4 dummy_worker /httpd/srclib/apr/threadproc/unix/thread.c:165
Thread T45 created by T0 here:
#1 apr_thread_create /httpd/srclib/apr/threadproc/unix/thread.c:235
#2 child_main /httpd/server/mpm/event/event.c:3055
#3 make_child /httpd/server/mpm/event/event.c:3168
#4 startup_children /httpd/server/mpm/event/event.c:3233
#5 event_run /httpd/server/mpm/event/event.c:3750
#6 ap_run_mpm /httpd/server/mpm_common.c:102
#7 main /httpd/server/main.c:882
Thread T46 created by T45 here:
#1 apr_thread_create /httpd/srclib/apr/threadproc/unix/thread.c:235
#2 start_threads /httpd/server/mpm/event/event.c:2813
#3 dummy_worker /httpd/srclib/apr/threadproc/unix/thread.c:165
In diagram form, the buggy sequence is:
T0
└► T45 (main thread)
├► T46 (child thread)
│ └─┐
│ │
└► T71 (listener thread)
├─┼─► allocated <struct recycled_pool> in a connection-specific
│ │ memory pool
│ └─► reacts to exit() due to main-socket shutdown by calling
│ ap_terminate() -- registered at startup via atexit() --
│ which in turn calls apr_pool_destroy(ap_pglobal);
│ at this point <struct recycled_pool> is freed
└───► tries to accept a new connection, and for this tries to
pop a pool from recycled list (queue_info->recycled_pools)
and gets a first pool <struct recycled_pool> and tries to
access its ->next field, but it was freed already!
Related source code:
*
https://github.com/apache/httpd/blob/e36237899d56fb457f43735b75e085ec3c320ecc/os/unix/unixd.c#L543
*
https://github.com/apache/httpd/blob/e36237899d56fb457f43735b75e085ec3c320ecc/server/mpm/event/event.c#L2228
*
https://github.com/apache/httpd/blob/e36237899d56fb457f43735b75e085ec3c320ecc/server/mpm/event/event.c#L2283
*
https://github.com/apache/httpd/blob/e36237899d56fb457f43735b75e085ec3c320ecc/server/mpm_fdqueue.c#L237
*
https://github.com/apache/httpd/blob/e36237899d56fb457f43735b75e085ec3c320ecc/server/mpm_fdqueue.c#L270
I don’t know of a simple fix for this issue. ideally, upon exit request, httpd
must interrupt the listener thread and wait until it truly finishes (in
particular, the listener thread must call ap_queue_info_free_idle_pools()
beforehand). Only then the on-exit logic can proceed with cleanups in
ap_terminate().
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
