Package: libapache2-mod-authnz-external
Version: 3.3.2-3
Severity: important
Tags: patch
Dear Debian maintainers,
I would like to report a subprocess-record lifetime bug in
mod_authnz_external. In exec_external(), the address of a
stack-allocated apr_proc_t is registered with the request pool for
cleanup. APR retains that address and may dereference it after
exec_external() has returned.
We found this while investigating intermittent SVN-over-HTTPS failures
on two Debian 13 hosting servers. A minimal proposed patch is attached.
Customer names, addresses, repository names and hostnames have been
omitted below.
1. Environment
Both production systems:
Debian GNU/Linux 13 (trixie), amd64
apache2 / apache2-bin: 2.4.71-1~deb13u1
libapache2-mod-authnz-external: 3.3.2-3
libapr1t64:amd64: 1.7.5-1
libapache2-mod-svn / libsvn1: 1.14.5-3
Apache MPM: event
Relevant loaded modules include authnz_external, dav_svn, authz_svn,
cgid, mod_perl and Passenger. The presence of mod_perl or Passenger is
not evidence that either is responsible; we have not established a
conflict involving those modules.
The unmodified mod_authnz_external.so on both hosts had SHA256:
35fe6ad5012969d79890bea70af20cfc456683f2e92479c6ce84505f791ea9dd
This matched the module extracted independently from Debian's official
3.3.2-3 amd64 package. dpkg -V reported no modification to the installed
package before our local fix.
2. Relevant configuration and observed workload
The SVN virtual hosts use an external PHP CLI authenticator via a pipe.
The relevant configuration is equivalent to:
DefineExternalAuth phplogin pipe /path/to/product/files/phpgitlogin.php
<Location /svn/>
DAV svn
SVNParentPath /path/to/product/files/svn
AuthType Basic
AuthName "SVN Access"
AuthBasicProvider external
AuthExternal phplogin
Require valid-user
AuthzSVNAccessFile /path/to/product/files/authz
AuthzSVNAnonymous off
</Location>
The affected client workload included updates and fresh checkouts
containing many files, with multiple requests/connections. Client user
agents included SVN 1.14.5 and SVN 1.15.0 on Windows, both using serf
1.3.10.
The original report followed migration to another hosting server. This
does not establish that the bug was introduced by a recent Apache
upgrade: the same source defect appears to have existed for some time,
and another server already had similar log entries.
3. Production symptoms
The customer reported these errors, depending on the attempt and
compression setting:
svn: E120104: ra_serf: An error occurred during decompression
svn: E120106: ra_serf: The server sent a truncated HTTP response body.
svn: E175009: The XML response contains invalid XML
svn: E130003: Malformed XML: no element found at line 22
An update sometimes processed only a few files before aborting.
Disabling HTTP compression did not resolve it. A client workaround
enabling bulk updates and extending the timeout helped; because both
options were changed together, that alone did not isolate the cause.
Representative server log messages, with identifying details redacted:
[authnz_external:error] (10)No child processes:
Could not get status from child process
[authnz_external:error] AuthExtern phplogin
[/path/to/product/files/phpgitlogin.php]: Failed (-5) for user
[redacted]
[auth_basic:error] AH01617: user [redacted]: authentication failure
for "/svn/[repository]/!svn/...": Password Mismatch
The "Password Mismatch" line follows the failure to collect the
authenticator's exit status; it should not be interpreted as proof of an
incorrect password.
We also observed:
[cgid:error] AH01239: cgid daemon process died, restarting
On the second server, scanning 15 retained Apache error-log files
spanning 1-15 November 2026 found 12 occurrences of "Could not get
status from child process", 12 corresponding "Failed (-5)" messages and
58 CGI daemon restart messages. No "Segmentation fault" messages were
found in that scan. These counts alone do not prove that every CGI
restart had the same cause.
4. Source-level problem
In exec_external() in mod_authnz_external.c, the relevant code is:
apr_pool_t *p = r->pool;
apr_proc_t proc;
...
rc = apr_proc_create(&proc, ...);
...
apr_pool_note_subprocess(p, &proc, APR_KILL_AFTER_TIMEOUT);
...
rc = apr_proc_wait(&proc, &status, &why, APR_WAIT);
...
return status;
The registration remains attached to the request pool after this
function returns, but proc is a local automatic variable whose lifetime
has ended.
In APR 1.7.5, apr_pool_note_subprocess() allocates a process_chain node
and stores the supplied pointer directly:
pc->proc = proc;
It does not copy the apr_proc_t. Later, free_proc_chain() uses pc->proc
with apr_proc_wait() and, depending on the result, apr_proc_kill().
Therefore request-pool cleanup can read an overwritten process record.
References:
https://sources.debian.org/src/libapache2-mod-authnz-external/3.3.2-2/mod_authnz_external.c/
https://github.com/apache/apr/blob/1.7.5/memory/unix/apr_pools.c
https://github.com/apache/apr/blob/1.7.5/threadproc/unix/proc.c
The source link above is for Debian revision -2; our installed package
is -3. We also examined the official 3.3.2 source archive and the
matching -3 binary. The binary disassembly confirms a stack-derived
address passed to apr_pool_note_subprocess(): r15 is set to rsp+0x60 at
0x27a1 and passed as the process-record argument at 0x27ec/0x27f2.
We additionally inspected the 3.3.3 upstream source archive available
from Debian's package pool; it still contains the same stack-variable
registration. We have not built or performed full Apache integration
tests against that version.
5. Consequences and isolated mechanism test
For example, if reused stack storage leaves pid equal to zero, Linux
waitpid(0, ...) can collect the status of another child in the same
process group. Its intended waiter may subsequently receive ECHILD. If
cleanup sees a running child and proceeds to kill(0, SIGTERM), the
signal targets processes in the caller's process group, subject to
normal permission checks.
This provides a plausible common mechanism for lost authenticator exit
statuses, worker/CGI termination and truncated HTTP responses. We did
not trace the original production requests at syscall level, so we
cannot claim to have captured that exact chain for each incident.
To validate the mechanism, we ran an isolated Python/ctypes harness
against the real Debian APR 1.7.5 shared library. The harness:
- creates an APR pool;
- creates an authentication-like child and registers its process record;
- collects that child's successful exit status;
- creates an unrelated child;
- deliberately overwrites the original record with zeros to model
reuse of
storage after the original function has returned;
- destroys the APR pool and checks the unrelated child's status and
signals.
Each scenario ran in a separate session/process group. This is a
deterministic test of the cleanup consequences of an overwritten record,
not a full reproduction of the original unpatched Apache request lifecycle.
Results:
Original pointer, unrelated child already exited:
APR cleanup collected the unrelated child's exit status.
Pool-owned copy, same scenario:
The unrelated child's exit status remained available to its owner.
Original pointer, unrelated child still running:
Cleanup signalled the isolated test process group and terminated
the child.
Pool-owned copy, same scenario:
No group signal occurred; the unrelated child remained running.
All four expectations passed. I can provide the harness and its output
if useful.
6. Proposed patch
The attachment changes the registration to retain a request-pool-owned copy:
apr_pool_note_subprocess(p, apr_pmemdup(p, &proc, sizeof(proc)),
APR_KILL_AFTER_TIMEOUT);
The existing synchronous apr_proc_wait() continues to operate on the
local proc. The copy retained by APR remains valid until request-pool
cleanup. The source already includes apr_strings.h, which declares
apr_pmemdup().
The patch targets the original 3.3.2 source and passed a dry-run
application check. It intentionally makes only this lifetime correction.
It does not change authentication decisions, access rules,
signal-handler handling, or the Apache MPM.
7. Build, deployment and verification
We built a local package version 3.3.2-3+local1 from Debian's 3.3.2-3
packaging in an isolated Debian 13 environment and deployed that
identical package on both servers, preserving backups of the original
package, module and Apache configuration.
The fixed module's SHA256 is:
829c3a8b2e076199abbe18e9b3d9ce1230a4f2514c8ad925838dc0d1ba799c2c
In the isolated Debian 13 environment, and again in a separate
localhost-only Apache test instance on the first production host, the
fixed module passed:
- 240 authenticated HTTP/HTTPS requests with up to 16 concurrent
workers;
- invalid-password checks over HTTP and HTTPS, both correctly
returning 401;
- four concurrent SVN checkouts over HTTPS, each containing 201 files,
including a 1 MiB binary file checked byte-for-byte;
- one SVN commit and three concurrent SVN updates over HTTPS;
- checks for ECHILD/Failed(-5), unexpected CGI restarts and
segmentation faults.
The temporary server test instance and repository were removed
afterwards. No customer repository was used for those writes.
On both production servers, Apache was fully restarted and the new
module was verified in the running process's mappings against the
installed file inode. Apache configuration tests passed. Repository
endpoints remained reachable with valid TLS and the expected HTTP 401
challenge when accessed without credentials. No new matching
process-error messages appeared during the short post-deployment checks.
A fresh checkout using the originally affected customer's real
credentials has not been performed, and long-term production observation
is still pending. Existing Passenger configuration warnings were present
before the change and were not addressed as part of this fix.
We subsequently enabled SVNAllowBulkUpdates Prefer as a separate
operational trial. The fixed-module integration tests on the first host
had already passed before that global setting was introduced. We are not
treating the bulk-update preference as a substitute for the lifetime fix
or as proof of improved performance.
Please review the proposed correction and consider an appropriate Debian
update and upstream forwarding. Please let me know if you would prefer
the process record to be allocated directly from the request pool
instead of copying it at registration, or if additional regression-test
coverage would be helpful.
Kind regards,
Sven Ruttloff
SaaS-Secure.com
--- a/mod_authnz_external.c
+++ b/mod_authnz_external.c
@@ -520,7 +520,10 @@
if (isdaemon) return 0;
- apr_pool_note_subprocess(p, &proc, APR_KILL_AFTER_TIMEOUT);
+ /* The request pool outlives this stack frame. APR retains the pointer,
+ * so its subprocess cleanup must refer to a pool-owned copy. */
+ apr_pool_note_subprocess(p, apr_pmemdup(p, &proc, sizeof(proc)),
+ APR_KILL_AFTER_TIMEOUT);
if (usepipein)
{