TL;DR
=====
Not a curl bug. The failure is an apr-util/OpenSSL 4 issue (LP: #2165311),
already fixed in apr-util 1.6.4-2ubuntu1 in stonking-proposed. The tests need
re-running with both triggers: curl/8.20.0-2ubuntu2 and
apr-util/1.6.4-2ubuntu1.
Long explanation
================
curl 8.20.0-2ubuntu2 is not the cause of this apache2 autopkgtest regression.
It only removes an accident that was hiding a pre-existing issue with
libaprutil1t64, fixed in stonking-proposed.
This is the understanding I have of the situation, if there are inaccuracies,
please do tell.
What fails
----------
apache2's test suite (debian/tests/run-test-suite.sh) loads every module in
/etc/apache2/mods-available, including mod_session_crypto. At startup
mod_session_crypto dlopen()s apr-util's OpenSSL driver and that fails:
[session_crypto:error] (20019)DSO load failed: AH01845: The crypto library
'openssl' could not be loaded:
/usr/lib/x86_64-linux-gnu/apr-util-1/apr_crypto_openssl-1.so:
undefined symbol: ENGINE_init (apr_crypto_openssl-1.so: 0)
[:emerg] AH00020: Configuration Failed, exiting
httpd exits during configuration, the perl-framework waits 60 seconds for the
already exited server, and run-test-suite_prefork/_event fail.
Root cause analysis
------------------
/usr/lib/x86_64-linux-gnu/apr-util-1/apr_crypto_openssl-1.so (libaprutil1t64
1.6.3-4ubuntu2) is linked against libcrypto.so.4 but still references the
OpenSSL ENGINE API, which libcrypto.so.4 no longer provides:
ENGINE_by_id, ENGINE_finish, ENGINE_free, ENGINE_init,
ENGINE_load_builtin_engines, ENGINE_register_all_complete
Undefined symbols are allowed in shared objects at link time, so this built and
shipped without any error. The references are also unversioned, so they can
bind to any matching definition that happens to be present in the process.
Curl saved the day until its libssl4 rebuild
--------------------------------------------
mod_md links libcurl, so libcurl is mapped into any httpd that loads
mod_md.
curl 8.20.0-2ubuntu1: libcurl.so.4 -> libcrypto.so.3. libcrypto.so.3 was
therefore mapped into httpd, and the unversioned ENGINE_* references in
apr_crypto_openssl-1.so bound to ENGINE_init@@OPENSSL_3.0.0 from it. The
driver loaded and the tests passed.
curl 8.20.0-2ubuntu2: libcurl.so.4 -> libcrypto.so.4. Nothing pulls OpenSSL 3
into the process any more, the ENGINE_* references stay unresolved, and the
dlopen fails.
The previous behaviour was not correct either: apr_crypto_openssl-1.so was
resolving ENGINE symbols from OpenSSL 3 while linking libcrypto.so.4 for
everything else, i.e. mixing two OpenSSL majors inside one library. The current
hard failure is bad, but it's better than the seemingly working situation we
had before.
Evidence
--------
The driver links OpenSSL 4:
(you can use libtree if you like shiny trees for deps)
$ objdump -p /usr/lib/x86_64-linux-gnu/apr-util-1/apr_crypto_openssl-1.so |
grep NEEDED
NEEDED libcrypto.so.4
NEEDED libc.so.6
but imports the ENGINE API, unversioned:
$ nm -D --undefined-only
/usr/lib/x86_64-linux-gnu/apr-util-1/apr_crypto_openssl-1.so | grep ENGINE
U ENGINE_by_id
U ENGINE_finish
U ENGINE_free
U ENGINE_init
U ENGINE_load_builtin_engines
U ENGINE_register_all_complete
libcrypto.so.4 does not provide it, libcrypto.so.3 does:
$ nm -D --defined-only /usr/lib/x86_64-linux-gnu/libcrypto.so.4 | grep -c
ENGINE_init
0
$ nm -D --defined-only /usr/lib/x86_64-linux-gnu/libcrypto.so.3 | grep
ENGINE_init
0000000000200630 T ENGINE_init@@OPENSSL_3.0.0
The driver is broken on its own, with no apache2 and no curl involved:
$ ldd -r /usr/lib/x86_64-linux-gnu/apr-util-1/apr_crypto_openssl-1.so | grep
ENGINE
undefined symbol: ENGINE_init
(/usr/lib/x86_64-linux-gnu/apr-util-1/apr_crypto_openssl-1.so)
undefined symbol: ENGINE_free
(/usr/lib/x86_64-linux-gnu/apr-util-1/apr_crypto_openssl-1.so)
undefined symbol: ENGINE_finish
(/usr/lib/x86_64-linux-gnu/apr-util-1/apr_crypto_openssl-1.so)
undefined symbol: ENGINE_register_all_complete
(/usr/lib/x86_64-linux-gnu/apr-util-1/apr_crypto_openssl-1.so)
undefined symbol: ENGINE_by_id
(/usr/lib/x86_64-linux-gnu/apr-util-1/apr_crypto_openssl-1.so)
undefined symbol: ENGINE_load_builtin_engines
(/usr/lib/x86_64-linux-gnu/apr-util-1/apr_crypto_openssl-1.so)
This last command reports the same unresolved symbols on a testbed running
curl 8.20.0-2ubuntu1, where the test suite passes. The DSO is broken in both
cases, but that was hidden by the curl version pulling SSL 3 symbols in the
process' scope.
As a reminder, the delta between the two curl builds on libcrypto/ssl:
$ objdump -p /usr/lib/x86_64-linux-gnu/libcurl.so.4 | grep -E
'libssl|libcrypto'
8.20.0-2ubuntu1:
NEEDED libssl.so.3
NEEDED libcrypto.so.3
required from libcrypto.so.3:
required from libssl.so.3:
8.20.0-2ubuntu2:
NEEDED libssl.so.4
NEEDED libcrypto.so.4
required from libssl.so.4:
required from libcrypto.so.4:
and how libcurl reaches httpd through mod_md.so:
$ objdump -p /usr/lib/apache2/modules/mod_md.so | grep NEEDED
NEEDED libcurl.so.4
[...]
Runtime confirmation on a testbed with curl 8.20.0-2ubuntu1:
$ LD_DEBUG=bindings LD_DEBUG_OUTPUT=/tmp/ld /usr/sbin/apache2 \
-d <workdir>/t -f <workdir>/t/conf/httpd.conf \
-D APACHE2 -D APACHE2_4 -D PERL_USEITHREADS
$ grep ENGINE_init /tmp/ld.*
Note: after the succeeding testbed finished and gave me a shell, the apache2
process started was still running, so I had to run
su tuserprefork -c '/usr/sbin/apache2 -d
/tmp/autopkgtest.zKl7tK/autopkgtest_tmp/perl-framework-mpm_prefork/t -f
/tmp/autopkgtest.zKl7tK/autopkgtest_tmp/perl-framework-
mpm_prefork/t/conf/httpd.conf -D APACHE2 -D APACHE2_4 -D
PERL_USEITHREADS -k stop'
to be able to collect the full ld debug:
root@autopkgtest-lxd-hvodue:/tmp/autopkgtest.zKl7tK/build.alj/src# su
tuserprefork -c 'LD_DEBUG=bindings LD_DEBUG_OUTPUT=/tmp/ld /usr/sbin/apache2
-d /tmp/autopkgtest.zKl7tK/autopkgtest_tmp/perl-framework-mpm_prefork/t -f
/tmp/autopkgtest.zKl7tK/autopkgtest_tmp/perl-framework-mpm_prefork/t/conf/httpd.conf
-D APACHE2 -D APACHE2_4 -D PERL_USEITHREADS'
[snip]
root@autopkgtest-lxd-hvodue:/tmp/autopkgtest.zKl7tK/build.alj/src# grep
ENGINE_init /tmp/ld.*
479515: binding file /usr/lib/x86_64-linux-gnu/libssl.so.3 [0] to
/usr/lib/x86_64-linux-gnu/libcrypto.so.3 [0]: normal symbol `ENGINE_init'
[OPENSSL_3.0.0]
479515: binding file /usr/lib/x86_64-linux-gnu/libcurl.so.4 [0] to
/usr/lib/x86_64-linux-gnu/libcrypto.so.3 [0]: normal symbol `ENGINE_init'
[OPENSSL_3.0.0]
479515: binding file /usr/lib/x86_64-linux-gnu/libcurl.so.4 [0] to
/usr/lib/x86_64-linux-gnu/libcrypto.so.3 [0]: normal symbol `ENGINE_init'
[OPENSSL_3.0.0]
479516: binding file
/usr/lib/x86_64-linux-gnu/apr-util-1/apr_crypto_openssl-1.so [0] to
/usr/lib/x86_64-linux-gnu/libcrypto.so.3 [0]: normal symbol `ENGINE_init'
libcurl pulls libcrypto.so.3 into the process, and apr_crypto_openssl-1.so then
binds ENGINE_init against it. Note the apr-util binding carries no version tag,
unlike libssl's and libcurl's: the reference is unversioned, which is what
allows a library linked against libcrypto.so.4 to resolve the symbol from
libcrypto.so.3.
On the failing testbed with curl 8.20.0-2ubuntu2:
root@autopkgtest-lxd-kyubkq:/tmp/autopkgtest.YkOjfb/build.L8c/src# su
tuserprefork -c 'LD_DEBUG=bindings LD_DEBUG_OUTPUT=/tmp/ld /usr/sbin/apache2
-d /tmp/autopkgtest.YkOjfb/autopkgtest_tmp/perl-framework-mpm_prefork/t -f
/tmp/autopkgtest.YkOjfb/autopkgtest_tmp/perl-framework-mpm_prefork/t/conf/httpd.conf
-D APACHE2 -D APACHE2_4 -D PERL_USEITHREADS'
[snip]
root@autopkgtest-lxd-kyubkq:/tmp/autopkgtest.YkOjfb/build.L8c/src# grep
ENGINE_init /tmp/ld.*
/tmp/ld.10772: 10773:
/usr/lib/x86_64-linux-gnu/apr-util-1/apr_crypto_openssl-1.so: error: symbol
lookup error: undefined symbol: ENGINE_init (fatal)
Conclusion
----------
This is already fixed. apr-util 1.6.4-2ubuntu1 in stonking-proposed removes the
remaining ENGINE references for OpenSSL 4 compatibility (LP: #2165311):
* Remove remaining ENGINE references for OpenSSL 4 compat (LP: #2165311)
- d/p/Remove_remaining_ENGINE_references.patch
apache2 itself is fine, its own modules link libcrypto.so.4 cleanly. curl
8.20.0-2ubuntu2 should not be held: the two simply need to migrate together.
It's worth to note that mod_session_crypto is not enabled by default. The test
suite loads every module in mods-available, which is why it surfaces here and
ordinary installs generally do not.
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/2166495
Title:
apache autopkgtests failing with new curl in stonking-proposed
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/apache2/+bug/2166495/+subscriptions
--
ubuntu-bugs mailing list
[email protected]
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
