Re: www/apache-httpd segfaults

[Pedro de Oliveira]

Hello,

Following your tips I built Apache with -O0 and the segfaults in
"mpm_prefork" no longer happen. Maybe its a gcc optimization issue.
I dont know which -O OpenBSD uses by default, but I will try it also with
-O1 and report back.

Now that it doesnt crash I fiddled a bit more with http2.

# gdb --args httpd2 -X
(gdb) run
Starting program: /usr/local/sbin/httpd2 -X
...

And when I do a "curl -v -http2 http://vhost.with.http2;

...
Program received signal SIGABRT, Aborted.
[Switching to thread 1016704]
0x05c7a6ae089a in thrkill () at :2
2   : No such file or directory.
in 
Current language:  auto; currently asm
(gdb) bt full
#0  0x05c7a6ae089a in thrkill () at :2
No locals.
#1  0x05c7a6adbf59 in *_libc_abort () at
/usr/src/lib/libc/stdlib/abort.c:52
mask = 4294967263
sa = {__sigaction_u = {__sa_handler = 0x1, __sa_sigaction = 0x1},
sa_mask = 49, sa_flags = 0}
#2  0x05c7e7fa336b in apr_table_copy () from
/usr/local/lib/libapr-1.so.5.0
No symbol table info available.
#3  0x05c805c85fa5 in h2_request_create_rec (req=0x5c840c98600,
conn=0x5c84dcb5400) at h2_request.c:361
r = (request_rec *) 0x5c76fbf2400
p = (apr_pool_t *) 0x5c84dcb5000
access_status = 200
#4  0x05c805c9b1cf in h2_task_process_request (req=0x5c840c98600,
c=0x5c84dcb5400) at h2_task.c:223
r = (request_rec *) 0x5c804d7bb40
cs = (conn_state_t *) 0x5c7dbcde2e0
#5  0x05c805c9b4dd in h2_task_process_conn (c=0x5c84dcb5400) at
h2_task.c:255
ctx = (h2_ctx *) 0x5c804d7bb40
#6  0x05c566b71f43 in ap_run_process_connection (c=0x5c84dcb5400) at
connection.c:41
pHook = (ap_LINK_process_connection_t *) 0x5c84d082600
n = 0
rv = -1
#7  0x05c566b7245d in ap_process_connection (c=0x5c84dcb5400,
csd=0x5c77f4a4280) at connection.c:213
rc = -2
#8  0x05c805c9ae43 in h2_task_do (task=0x5c7dbcdee80,
worker=0x5c79f7f8880) at h2_task.c:186
status = 0
#9  0x05c805ca1386 in execute (thread=0x5c82c394800,
wctx=0x5c79f7f8880) at h2_worker.c:58
worker = (h2_worker *) 0x5c79f7f8880
status = 0
m = (h2_mplx *) 0x5c84dcb5300
#10 0x05c845c2c77e in _rthread_start (v=Variable "v" is not available.
) at /usr/src/lib/librthread/rthread.c:145
retval = Variable "retval" is not available.
(gdb)

Any hints?

On Sat, Jan 16, 2016 at 6:16 AM, David CARLIER  wrote:

> Hi,
>
> all depends on the configuration. Also, what would be possible is to
> try to compile apache with debug symbols (-O0) to have better trace of
> the bug (the bug might occur line 732 of server/mpm/prefork/prefork.c
> but it is just guess).
>
> Regards.
>
> On 15 January 2016 at 23:41, Stuart Henderson  wrote:
> > I've disabled mod_http2 from being packaged for now, not sure how
> > to debug it further at present.
> >
> >
> > On 2016/01/14 18:58, Pedro de Oliveira wrote:
> >> Hi there,
> >>
> >> Yesterday I was trying to play with the http2 module on Apache from
> >> ports, as I saw it was added to the ports file list. The problem is
> >> that the module isnt working correctly.
> >>
> >> If you try it on a http vhost, it simply returns the page via the
> >> normal way (http).
> >> And if you enable it on a https vhost, the vhost stops working (doesnt
> >> serve anything).
> >>
> >> So I went to https://httpd.apache.org/dev/debugging.html , and saw the
> >> instructions on how to debug Apache, and ran httpd2 on gdb with the -X
> >> option.
> >>
> >> After a few requests, httpd2 segfaults, but only when using the
> >> "mpm_prefork" module. With "mpm_event" or "mpm_worker" it doesnt
> >> crash.
> >>
> >> # gdb --args httpd2 -X
> >> (gdb) run
> >> Starting program: /usr/local/sbin/httpd2 -X
> >> AH00557: httpd2: apr_sockaddr_info_get() failed for web.my.domain
> >> AH00558: httpd2: Could not reliably determine the server's fully
> >> qualified domain name, using 127.0.0.1. Set the 'ServerName' directive
> >> globally to suppress this message
> >>
> >> Program received signal SIGSEGV, Segmentation fault.
> >> 0x14a785962a1b in ap_mpm_pod_check () from /usr/local/sbin/httpd2
> >> (gdb) bt
> >> #0  0x14a785962a1b in ap_mpm_pod_check () from
> /usr/local/sbin/httpd2
> >> #1  0x14aa29a81f9e in child_main ()
> >>from /usr/local/lib/apache2/mod_mpm_prefork.so
> >> #2  0x14aa29a82321 in make_child ()
> >>from /usr/local/lib/apache2/mod_mpm_prefork.so
> >> #3  0x14aa29a833cf in prefork_run ()
> >>from /usr/local/lib/apache2/mod_mpm_prefork.so
> >> #4  0x14a785937d12 in ap_run_mpm () from /usr/local/sbin/httpd2
> >> #5  0x14a7859318b3 in main () from /usr/local/sbin/httpd2
> >>
> >> And if you run httpd2 -X and try to access an SSL vhost it also crashes.
> >>
> >> Program received signal SIGPIPE, Broken pipe.
> >> [Switching to thread 1003084]
> >> 0x1ff81dfaf1aa in writev () at :2
> >> 2   : No such file or directory.
> >> in 
> >> Current 

Re: www/apache-httpd segfaults

[David CARLIER]

bug reproduced, happens pretty quickly once I stressed httpd2 with siege ...

On 16 January 2016 at 19:48, Pedro de Oliveira  wrote:
> Ignore what I said in the earlier email about "mpm_prefork", I wasnt
> using it. Sorry.
>
> It still crashes after a few requests. This is the backtrace from
> apache build with -O0 and -ggdb.
>
> # gdb --args httpd2 -X
> (gdb) run
> Starting program: /usr/local/sbin/httpd2 -X
> ...
> Program received signal SIGSEGV, Segmentation fault.
> 0x18a9b2875638 in ap_mpm_pod_check (pod=0x0) at mpm_unix.c:459
> 459 rv = apr_file_read(pod->pod_in, , );
> (gdb) bt full
> #0  0x18a9b2875638 in ap_mpm_pod_check (pod=0x0) at mpm_unix.c:459
> c = -1 '▒'
> len = 1
> rv = 2
> #1  0x18ac9892d3bb in child_main (child_num_arg=0, child_bucket=0)
> at prefork.c:732
> current_conn = (conn_rec *) 0x18abbed08f00
> csd = (void *) 0x18abb4508000
> thd = (apr_thread_t *) 0x18ac9e058b40
> osthd = 0x18abeee85920
> ptrans = (apr_pool_t *) 0x18abc3acce00
> allocator = (apr_allocator_t *) 0x18abe0001700
> status = 0
> i = -1
> lr = (ap_listen_rec *) 0x18abfdd7be00
> pollset = (apr_pollset_t *) 0x18abb4509e00
> sbh = (ap_sb_handle_t *) 0x18abed2a6e40
> bucket_alloc = (apr_bucket_alloc_t *) 0x18ac537a7028
> last_poll_idx = 1
> lockfile = 0x7f7d4d20 ""
> #2  0x18ac9892d4d5 in make_child (s=0x18ac45b90d00, slot=0,
> bucket=0) at prefork.c:767
> pid = 1
> #3  0x18ac9892dc2d in prefork_run (_pconf=0x18abc7307d00,
> plog=0x18abc7307700, s=0x18ac45b90d00) at prefork.c:979
> index = 6316
> remaining_children_to_start = 2134903585
> i = 6315
> #4  0x18a9b283a8b7 in ap_run_mpm (pconf=0x18abc7307d00,
> plog=0x18abc7307700, s=0x18ac45b90d00) at mpm_common.c:94
> pHook = (ap_LINK_mpm_t *) 0x18abd5664000
> n = 0
> rv = -1
> #5  0x18a9b28324a3 in main (argc=2, argv=0x7f7d5158) at main.c:777
> c = 88 'X'
> showcompile = 0
> showdirectives = 0
> confname = 0x18a9b299083b "/etc/apache2/httpd2.conf"
> def_server_root = 0x18a9b2990854 "/var/www"
> temp_error_log = 0x0
> error = 0x0
> process = (process_rec *) 0x18ac9cb312c0
> pconf = (apr_pool_t *) 0x18abc7307d00
> plog = (apr_pool_t *) 0x18abc7307700
> ptemp = (apr_pool_t *) 0x18abbb0bb800
> pcommands = (apr_pool_t *) 0x18ac5cf89500
> opt = (apr_getopt_t *) 0x18ac48aed600
> rv = 0
> mod = (module **) 0x18a9b2cac518
> opt_arg = 0x0
> signal_server = (apr_OFN_ap_signal_server_t *) 0x18a9b2875ee5
> 
>
>
>
> On Sat, Jan 16, 2016 at 7:28 PM, Michael McConville  wrote:
>> Pedro de Oliveira wrote:
>>> Following your tips I built Apache with -O0 and the segfaults in
>>> "mpm_prefork" no longer happen. Maybe its a gcc optimization issue. I
>>> dont know which -O OpenBSD uses by default, but I will try it also
>>> with -O1 and report back.
>>
>> Remember, compiler optimizations can worsen the impact of undefined
>> behavior. That seems more likely than a compiler bug to me.
>

Re: www/apache-httpd segfaults

[Michael McConville]

Pedro de Oliveira wrote:
> Following your tips I built Apache with -O0 and the segfaults in
> "mpm_prefork" no longer happen. Maybe its a gcc optimization issue. I
> dont know which -O OpenBSD uses by default, but I will try it also
> with -O1 and report back.

Remember, compiler optimizations can worsen the impact of undefined
behavior. That seems more likely than a compiler bug to me.

Re: www/apache-httpd segfaults

[Pedro de Oliveira]

Ignore what I said in the earlier email about "mpm_prefork", I wasnt
using it. Sorry.

It still crashes after a few requests. This is the backtrace from
apache build with -O0 and -ggdb.

# gdb --args httpd2 -X
(gdb) run
Starting program: /usr/local/sbin/httpd2 -X
...
Program received signal SIGSEGV, Segmentation fault.
0x18a9b2875638 in ap_mpm_pod_check (pod=0x0) at mpm_unix.c:459
459 rv = apr_file_read(pod->pod_in, , );
(gdb) bt full
#0  0x18a9b2875638 in ap_mpm_pod_check (pod=0x0) at mpm_unix.c:459
c = -1 '▒'
len = 1
rv = 2
#1  0x18ac9892d3bb in child_main (child_num_arg=0, child_bucket=0)
at prefork.c:732
current_conn = (conn_rec *) 0x18abbed08f00
csd = (void *) 0x18abb4508000
thd = (apr_thread_t *) 0x18ac9e058b40
osthd = 0x18abeee85920
ptrans = (apr_pool_t *) 0x18abc3acce00
allocator = (apr_allocator_t *) 0x18abe0001700
status = 0
i = -1
lr = (ap_listen_rec *) 0x18abfdd7be00
pollset = (apr_pollset_t *) 0x18abb4509e00
sbh = (ap_sb_handle_t *) 0x18abed2a6e40
bucket_alloc = (apr_bucket_alloc_t *) 0x18ac537a7028
last_poll_idx = 1
lockfile = 0x7f7d4d20 ""
#2  0x18ac9892d4d5 in make_child (s=0x18ac45b90d00, slot=0,
bucket=0) at prefork.c:767
pid = 1
#3  0x18ac9892dc2d in prefork_run (_pconf=0x18abc7307d00,
plog=0x18abc7307700, s=0x18ac45b90d00) at prefork.c:979
index = 6316
remaining_children_to_start = 2134903585
i = 6315
#4  0x18a9b283a8b7 in ap_run_mpm (pconf=0x18abc7307d00,
plog=0x18abc7307700, s=0x18ac45b90d00) at mpm_common.c:94
pHook = (ap_LINK_mpm_t *) 0x18abd5664000
n = 0
rv = -1
#5  0x18a9b28324a3 in main (argc=2, argv=0x7f7d5158) at main.c:777
c = 88 'X'
showcompile = 0
showdirectives = 0
confname = 0x18a9b299083b "/etc/apache2/httpd2.conf"
def_server_root = 0x18a9b2990854 "/var/www"
temp_error_log = 0x0
error = 0x0
process = (process_rec *) 0x18ac9cb312c0
pconf = (apr_pool_t *) 0x18abc7307d00
plog = (apr_pool_t *) 0x18abc7307700
ptemp = (apr_pool_t *) 0x18abbb0bb800
pcommands = (apr_pool_t *) 0x18ac5cf89500
opt = (apr_getopt_t *) 0x18ac48aed600
rv = 0
mod = (module **) 0x18a9b2cac518
opt_arg = 0x0
signal_server = (apr_OFN_ap_signal_server_t *) 0x18a9b2875ee5




On Sat, Jan 16, 2016 at 7:28 PM, Michael McConville  wrote:
> Pedro de Oliveira wrote:
>> Following your tips I built Apache with -O0 and the segfaults in
>> "mpm_prefork" no longer happen. Maybe its a gcc optimization issue. I
>> dont know which -O OpenBSD uses by default, but I will try it also
>> with -O1 and report back.
>
> Remember, compiler optimizations can worsen the impact of undefined
> behavior. That seems more likely than a compiler bug to me.

Re: www/apache-httpd segfaults

[Stuart Henderson]

I've disabled mod_http2 from being packaged for now, not sure how
to debug it further at present.


On 2016/01/14 18:58, Pedro de Oliveira wrote:
> Hi there,
> 
> Yesterday I was trying to play with the http2 module on Apache from
> ports, as I saw it was added to the ports file list. The problem is
> that the module isnt working correctly.
> 
> If you try it on a http vhost, it simply returns the page via the
> normal way (http).
> And if you enable it on a https vhost, the vhost stops working (doesnt
> serve anything).
> 
> So I went to https://httpd.apache.org/dev/debugging.html , and saw the
> instructions on how to debug Apache, and ran httpd2 on gdb with the -X
> option.
> 
> After a few requests, httpd2 segfaults, but only when using the
> "mpm_prefork" module. With "mpm_event" or "mpm_worker" it doesnt
> crash.
> 
> # gdb --args httpd2 -X
> (gdb) run
> Starting program: /usr/local/sbin/httpd2 -X
> AH00557: httpd2: apr_sockaddr_info_get() failed for web.my.domain
> AH00558: httpd2: Could not reliably determine the server's fully
> qualified domain name, using 127.0.0.1. Set the 'ServerName' directive
> globally to suppress this message
> 
> Program received signal SIGSEGV, Segmentation fault.
> 0x14a785962a1b in ap_mpm_pod_check () from /usr/local/sbin/httpd2
> (gdb) bt
> #0  0x14a785962a1b in ap_mpm_pod_check () from /usr/local/sbin/httpd2
> #1  0x14aa29a81f9e in child_main ()
>from /usr/local/lib/apache2/mod_mpm_prefork.so
> #2  0x14aa29a82321 in make_child ()
>from /usr/local/lib/apache2/mod_mpm_prefork.so
> #3  0x14aa29a833cf in prefork_run ()
>from /usr/local/lib/apache2/mod_mpm_prefork.so
> #4  0x14a785937d12 in ap_run_mpm () from /usr/local/sbin/httpd2
> #5  0x14a7859318b3 in main () from /usr/local/sbin/httpd2
> 
> And if you run httpd2 -X and try to access an SSL vhost it also crashes.
> 
> Program received signal SIGPIPE, Broken pipe.
> [Switching to thread 1003084]
> 0x1ff81dfaf1aa in writev () at :2
> 2   : No such file or directory.
> in 
> Current language:  auto; currently asm
> (gdb) bt
> #0  0x1ff81dfaf1aa in writev () at :2
> #1  0x1ff7a7eec2ed in writev (fd=19, iov=0x1ff7789a7780, iovcnt=1)
> at /usr/src/lib/librthread/rthread_cancel.c:621
> #2  0x1ff75d837311 in apr_socket_sendv () from
> /usr/local/lib/libapr-1.so.5.0
> #3  0x1ff53a44f3b8 in writev_nonblocking () from /usr/local/sbin/httpd2
> #4  0x1ff53a44f6e3 in send_brigade_nonblocking () from
> /usr/local/sbin/httpd2
> #5  0x1ff53a44fbc9 in ap_core_output_filter () from /usr/local/sbin/httpd2
> #6  0x1ff7e916011c in bio_filter_out_pass () from
> /usr/local/lib/apache2/mod_ssl.so
> #7  0x1ff7e9161955 in bio_filter_out_ctrl () from
> /usr/local/lib/apache2/mod_ssl.so
> #8  0x1ff7e9171d42 in modssl_smart_shutdown () from
> /usr/local/lib/apache2/mod_ssl.so
> #9  0x1ff7e916059c in ssl_filter_io_shutdown () from
> /usr/local/lib/apache2/mod_ssl.so
> #10 0x1ff7e9161dcf in ssl_io_filter_output () from
> /usr/local/lib/apache2/mod_ssl.so
> #11 0x1ff7e9162498 in ssl_io_filter_coalesce () from
> /usr/local/lib/apache2/mod_ssl.so
> #12 0x1ff53a45f72a in ap_shutdown_conn () from /usr/local/sbin/httpd2
> #13 0x1ff53a45f798 in ap_start_lingering_close () from
> /usr/local/sbin/httpd2
> #14 0x1ff8185d03a4 in worker_thread () from
> /usr/local/lib/apache2/mod_mpm_event.so
> #15 0x1ff7a7ee977e in _rthread_start (v=Variable "v" is not available.
> ) at /usr/src/lib/librthread/rthread.c:145
> #16 0x1ff81df9952b in __tfork_thread () at
> /usr/src/lib/libc/arch/amd64/sys/tfork_thread.S:75
> #17 0x in ?? ()
> 
> I dont know if this is expected on OpenBSD or already known, if it is,
> please ignore this message.
> 
> This is making debugging Apache to figure out why http2 isnt working 
> impossible.
> 
> Regards.
> 

www/apache-httpd segfaults

[Pedro de Oliveira]

Hi there,

Yesterday I was trying to play with the http2 module on Apache from
ports, as I saw it was added to the ports file list. The problem is
that the module isnt working correctly.

If you try it on a http vhost, it simply returns the page via the
normal way (http).
And if you enable it on a https vhost, the vhost stops working (doesnt
serve anything).

So I went to https://httpd.apache.org/dev/debugging.html , and saw the
instructions on how to debug Apache, and ran httpd2 on gdb with the -X
option.

After a few requests, httpd2 segfaults, but only when using the
"mpm_prefork" module. With "mpm_event" or "mpm_worker" it doesnt
crash.

# gdb --args httpd2 -X
(gdb) run
Starting program: /usr/local/sbin/httpd2 -X
AH00557: httpd2: apr_sockaddr_info_get() failed for web.my.domain
AH00558: httpd2: Could not reliably determine the server's fully
qualified domain name, using 127.0.0.1. Set the 'ServerName' directive
globally to suppress this message

Program received signal SIGSEGV, Segmentation fault.
0x14a785962a1b in ap_mpm_pod_check () from /usr/local/sbin/httpd2
(gdb) bt
#0  0x14a785962a1b in ap_mpm_pod_check () from /usr/local/sbin/httpd2
#1  0x14aa29a81f9e in child_main ()
   from /usr/local/lib/apache2/mod_mpm_prefork.so
#2  0x14aa29a82321 in make_child ()
   from /usr/local/lib/apache2/mod_mpm_prefork.so
#3  0x14aa29a833cf in prefork_run ()
   from /usr/local/lib/apache2/mod_mpm_prefork.so
#4  0x14a785937d12 in ap_run_mpm () from /usr/local/sbin/httpd2
#5  0x14a7859318b3 in main () from /usr/local/sbin/httpd2

And if you run httpd2 -X and try to access an SSL vhost it also crashes.

Program received signal SIGPIPE, Broken pipe.
[Switching to thread 1003084]
0x1ff81dfaf1aa in writev () at :2
2   : No such file or directory.
in 
Current language:  auto; currently asm
(gdb) bt
#0  0x1ff81dfaf1aa in writev () at :2
#1  0x1ff7a7eec2ed in writev (fd=19, iov=0x1ff7789a7780, iovcnt=1)
at /usr/src/lib/librthread/rthread_cancel.c:621
#2  0x1ff75d837311 in apr_socket_sendv () from
/usr/local/lib/libapr-1.so.5.0
#3  0x1ff53a44f3b8 in writev_nonblocking () from /usr/local/sbin/httpd2
#4  0x1ff53a44f6e3 in send_brigade_nonblocking () from
/usr/local/sbin/httpd2
#5  0x1ff53a44fbc9 in ap_core_output_filter () from /usr/local/sbin/httpd2
#6  0x1ff7e916011c in bio_filter_out_pass () from
/usr/local/lib/apache2/mod_ssl.so
#7  0x1ff7e9161955 in bio_filter_out_ctrl () from
/usr/local/lib/apache2/mod_ssl.so
#8  0x1ff7e9171d42 in modssl_smart_shutdown () from
/usr/local/lib/apache2/mod_ssl.so
#9  0x1ff7e916059c in ssl_filter_io_shutdown () from
/usr/local/lib/apache2/mod_ssl.so
#10 0x1ff7e9161dcf in ssl_io_filter_output () from
/usr/local/lib/apache2/mod_ssl.so
#11 0x1ff7e9162498 in ssl_io_filter_coalesce () from
/usr/local/lib/apache2/mod_ssl.so
#12 0x1ff53a45f72a in ap_shutdown_conn () from /usr/local/sbin/httpd2
#13 0x1ff53a45f798 in ap_start_lingering_close () from
/usr/local/sbin/httpd2
#14 0x1ff8185d03a4 in worker_thread () from
/usr/local/lib/apache2/mod_mpm_event.so
#15 0x1ff7a7ee977e in _rthread_start (v=Variable "v" is not available.
) at /usr/src/lib/librthread/rthread.c:145
#16 0x1ff81df9952b in __tfork_thread () at
/usr/src/lib/libc/arch/amd64/sys/tfork_thread.S:75
#17 0x in ?? ()

I dont know if this is expected on OpenBSD or already known, if it is,
please ignore this message.

This is making debugging Apache to figure out why http2 isnt working impossible.

Regards.