Hi, I’m currently working on a new php formula in the Homebrew package manager project where we have noticed that httpd crashes on macOS High Sierra under certain conditions. My question now is if this is a bug in Apache httpd that I should file in the bug tracker or if you can point me in the right direction to get to the bottom of this.
In short what happens is that httpd (loading libphp7) crashes when the php snmp extension in php is loaded with a reference to boringssl but when the snmp extension is omitted it doesn’t crash. # httpd -X [1] 52393 abort httpd -X # Using the php binary alone with the snmp extension enabled causes no issues as far as we’ve been able to tell. The stack trace from the crash looks like this Process: httpd [52393] Path: /usr/local/Cellar/httpd/2.4.29_1/bin/httpd Identifier: httpd Version: 0 Code Type: X86-64 (Native) Parent Process: zsh [541] Responsible: httpd [52393] User ID: 501 Date/Time: 2018-02-14 10:49:57.591 +0100 OS Version: Mac OS X 10.13.3 (17D47) Report Version: 12 Anonymous UUID: AC6370DE-1692-0BA1-08A2-BBE6153A8410 Sleep/Wake UUID: F545ED14-2DF3-4149-89EA-86A67EC93AB1 Time Awake Since Boot: 85000 seconds Time Since Wake: 84000 seconds System Integrity Protection: enabled Crashed Thread: 0 Dispatch queue: com.apple.main-thread Exception Type: EXC_CRASH (SIGABRT) Exception Codes: 0x0000000000000000, 0x0000000000000000 Exception Note: EXC_CORPSE_NOTIFY Application Specific Information: Assertion failed: (ctx), function digest_update, file /BuildRoot/Library/Caches/com.apple.xbs/Sources/boringssl/boringssl-109.40.1/apple/crypto/digests.c, line 49. Thread 0 Crashed:: Dispatch queue: com.apple.main-thread 0 libsystem_kernel.dylib 0x00007fff5f670e3e __pthread_kill + 10 1 libsystem_pthread.dylib 0x00007fff5f7af150 pthread_kill + 333 2 libsystem_c.dylib 0x00007fff5f5cd312 abort + 127 3 libsystem_c.dylib 0x00007fff5f595368 __assert_rtn + 320 4 libboringssl.dylib 0x00007fff5d4b4ddc digest_update + 51 5 libcrypto.35.dylib 0x00007fff5db0fe08 EVP_DigestInit_ex + 920 6 libcrypto.35.dylib 0x00007fff5db0fa6a EVP_DigestInit + 42 7 libnetsnmp.30.dylib 0x000000010c089b8b sc_hash + 202 8 libnetsnmp.30.dylib 0x000000010c088f57 hash_engineID + 116 9 libnetsnmp.30.dylib 0x000000010c088d7f search_enginetime_list + 40 10 libnetsnmp.30.dylib 0x000000010c08909b set_enginetime + 61 11 libnetsnmp.30.dylib 0x000000010c088a82 init_snmpv3_post_config + 150 12 libnetsnmp.30.dylib 0x000000010c08a7a5 snmp_call_callbacks + 432 13 snmp.so 0x000000010bdcf1b0 zm_startup_snmp + 59 14 libphp7.so 0x0000000108ee023a zend_startup_module_ex + 305 15 libphp7.so 0x0000000108ee05bb zend_startup_module_zval + 12 16 libphp7.so 0x0000000108eec6e1 zend_hash_apply + 201 17 libphp7.so 0x0000000108ee04d8 zend_startup_modules + 47 18 libphp7.so 0x0000000108e82b0a php_module_startup + 2215 19 libphp7.so 0x0000000108f9b426 php_apache2_startup + 21 20 libphp7.so 0x0000000108f9ad33 php_apache_server_startup + 105 21 httpd 0x00000001088bd50b ap_run_post_config + 69 22 httpd 0x00000001088c4c2c main + 2076 23 libdyld.dylib 0x00007fff5f521115 start + 1 Thread 1: 0 libsystem_kernel.dylib 0x00007fff5f671562 __workq_kernreturn + 10 1 libsystem_pthread.dylib 0x00007fff5f7ac06a _pthread_wqthread + 1035 2 libsystem_pthread.dylib 0x00007fff5f7abc4d start_wqthread + 13 Thread 2: 0 libsystem_kernel.dylib 0x00007fff5f671562 __workq_kernreturn + 10 1 libsystem_pthread.dylib 0x00007fff5f7ac26f _pthread_wqthread + 1552 2 libsystem_pthread.dylib 0x00007fff5f7abc4d start_wqthread + 13 Thread 3: 0 libsystem_pthread.dylib 0x00007fff5f7abc40 start_wqthread + 0 1 ??? 0x000000005e943d01 0 + 1586773249 Thread 0 crashed with X86 Thread State (64-bit): rax: 0x0000000000000000 rbx: 0x00007fff9857a340 rcx: 0x00007ffee7343998 rdx: 0x0000000000000000 rdi: 0x0000000000000307 rsi: 0x0000000000000006 rbp: 0x00007ffee73439d0 rsp: 0x00007ffee7343998 r8: 0x0000000000000000 r9: 0x0000000000000000 r10: 0x0000000000000000 r11: 0x0000000000000206 r12: 0x0000000000000307 r13: 0x000000010bdff000 r14: 0x0000000000000006 r15: 0x000000000000002d rip: 0x00007fff5f670e3e rfl: 0x0000000000000206 cr2: 0x00007fff98558148 Logical CPU: 0 Error Code: 0x02000148 Trap Number: 133 (this has been shortened and the full outtake can be found in the gist below) Looking at that stack trace it seems that there should be an issue with boringssl under macOS however none of the components httpd, php nor the snmp extension has been linked against boringssl or the macos provided ssl binaries. The shared libraries that the snmp extension uses looks like this # otool -L /usr/local/opt/php/lib/php/20170718/snmp.so /usr/local/opt/php/lib/php/20170718/snmp.so: /usr/local/opt/net-snmp/lib/libnetsnmp.30.dylib (compatibility version 31.0.0, current version 31.3.0) /usr/local/opt/openssl/lib/libcrypto.1.0.0.dylib (compatibility version 1.0.0, current version 1.0.0) /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1252.0.0) (brew linkage is a homebrew command to list linked libraries across all packaged components) # brew linkage httpd System libraries: /usr/lib/libSystem.B.dylib /usr/lib/libexpat.1.dylib /usr/lib/libiconv.2.dylib /usr/lib/libxml2.2.dylib /usr/lib/libz.1.dylib Homebrew libraries: /usr/local/opt/apr/libexec/lib/libapr-1.0.dylib (apr) /usr/local/opt/apr-util/libexec/lib/libaprutil-1.0.dylib (apr-util) /usr/local/opt/nghttp2/lib/libnghttp2.14.dylib (nghttp2) /usr/local/opt/openssl/lib/libcrypto.1.0.0.dylib (openssl) /usr/local/opt/openssl/lib/libssl.1.0.0.dylib (openssl) /usr/local/opt/pcre/lib/libpcre.1.dylib (pcre) # brew linkage php System libraries: /System/Library/Frameworks/Kerberos.framework/Versions/A/Kerberos /System/Library/Frameworks/LDAP.framework/Versions/A/LDAP /usr/lib/libSystem.B.dylib /usr/lib/libbz2.1.0.dylib /usr/lib/libc++.1.dylib /usr/lib/libcurl.4.dylib /usr/lib/libedit.3.dylib /usr/lib/libexpat.1.dylib /usr/lib/libexslt.0.dylib /usr/lib/libiconv.2.dylib /usr/lib/libicucore.A.dylib /usr/lib/libncurses.5.4.dylib /usr/lib/libpam.2.dylib /usr/lib/libresolv.9.dylib /usr/lib/libtidy.A.dylib /usr/lib/libxml2.2.dylib /usr/lib/libxslt.1.dylib /usr/lib/libz.1.dylib Homebrew libraries: /usr/local/opt/apr/libexec/lib/libapr-1.0.dylib (apr) /usr/local/opt/apr-util/libexec/lib/libaprutil-1.0.dylib (apr-util) /usr/local/opt/argon2/lib/libargon2.1.dylib (argon2) /usr/local/opt/aspell/lib/libaspell.15.dylib (aspell) /usr/local/opt/aspell/lib/libpspell.15.dylib (aspell) /usr/local/opt/freetds/lib/libsybdb.5.dylib (freetds) /usr/local/opt/freetype/lib/libfreetype.6.dylib (freetype) /usr/local/opt/gettext/lib/libintl.8.dylib (gettext) /usr/local/opt/gmp/lib/libgmp.10.dylib (gmp) /usr/local/opt/icu4c/lib/libicudata.60.2.dylib (icu4c) /usr/local/opt/icu4c/lib/libicui18n.60.dylib (icu4c) /usr/local/opt/icu4c/lib/libicuio.60.dylib (icu4c) /usr/local/opt/icu4c/lib/libicuuc.60.dylib (icu4c) /usr/local/opt/jpeg/lib/libjpeg.9.dylib (jpeg) /usr/local/opt/libpng/lib/libpng16.16.dylib (libpng) /usr/local/opt/libpq/lib/libpq.5.dylib (libpq) /usr/local/opt/libsodium/lib/libsodium.23.dylib (libsodium) /usr/local/opt/libzip/lib/libzip.5.dylib (libzip) /usr/local/opt/net-snmp/lib/libnetsnmp.30.dylib (net-snmp) /usr/local/opt/openssl/lib/libcrypto.1.0.0.dylib (openssl) /usr/local/opt/openssl/lib/libssl.1.0.0.dylib (openssl) /usr/local/opt/unixodbc/lib/libodbc.2.dylib (unixodbc) /usr/local/opt/webp/lib/libwebp.7.dylib (webp) ➜ homebrew-core git:(master) ✗ brew linkage net-snmp System libraries: /usr/lib/libSystem.B.dylib Homebrew libraries: /usr/local/Cellar/net-snmp/5.7.3/lib/libnetsnmp.30.dylib (net-snmp) /usr/local/Cellar/net-snmp/5.7.3/lib/libnetsnmpagent.30.dylib (net-snmp) /usr/local/Cellar/net-snmp/5.7.3/lib/libnetsnmpmibs.30.dylib (net-snmp) /usr/local/Cellar/net-snmp/5.7.3/lib/libnetsnmptrapd.30.dylib (net-snmp) /usr/local/opt/openssl/lib/libcrypto.1.0.0.dylib (openssl) So concluding that boringssl in not actively linked against any component involved It looks like httpd seems to be dynamically linking against the incorrect library which therefore causes the system to crash. The workaround that has been found so far is to statically link the Apache httpd SSL module using --enable-mods-static=ssl which make the system not crash but it doesn’t solve the incorrect dynamic loading. The httpd binary is built with the following options ./configure --enable-layout=Slackware-FHS --prefix=/usr/local/Cellar/httpd/2.4.29_1 --sbindir=/usr/local/Cellar/httpd/2.4.29_1/bin --mandir=/usr/local/Cellar/httpd/2.4.29_1/share/man --sysconfdir=/usr/local/etc/httpd --datadir=/usr/local/var/www --localstatedir=/usr/local/var --enable-mpms-shared=all --enable-mods-shared=all --enable-cgi --enable-pie --enable-suexec --with-suexec-bin=/usr/local/opt/httpd/bin/suexec --with-suexec-caller=_www --with-port=8080 --with-sslport=8443 --with-apr=/usr/local/opt/apr --with-apr-util=/usr/local/opt/apr-util --with-mpm=prefork --with-nghttp2=/usr/local/opt/nghttp2 --with-ssl=/usr/local/opt/openssl --with-pcre=/usr/local/opt/pcre The Erlang project seems to have run into a similar issue and have a solution for their software which can be found here https://bugs.erlang.org/browse/ERL-439 Full stack trace report can be found here https://gist.github.com/javian/1c530a2daec748cafe32a47ea2512e94 So to reiterate my question - do you think this is this a bug in Apache httpd that I should file in the bug tracker or can you point me in another directions to find the root cause of this issue ? I’m happy to provide further information and test whatever I can to help troubleshoot this. Best regards, Jan