[Bug 59840] Segmentation fault during the request processing

[bugzilla]

https://bz.apache.org/bugzilla/show_bug.cgi?id=59840

Eric Covener  changed:

   What|Removed |Added

 Resolution|--- |CLOSED
 Status|NEW |RESOLVED

-- 
You are receiving this mail because:
You are the assignee for the bug.
-
To unsubscribe, e-mail: bugs-unsubscr...@httpd.apache.org
For additional commands, e-mail: bugs-h...@httpd.apache.org

[Bug 59840] Segmentation fault during the request processing

[bugzilla]

https://bz.apache.org/bugzilla/show_bug.cgi?id=59840

juniorolald...@gmail.com changed:

   What|Removed |Added

 Status|NEEDINFO|NEW

-- 
You are receiving this mail because:
You are the assignee for the bug.
-
To unsubscribe, e-mail: bugs-unsubscr...@httpd.apache.org
For additional commands, e-mail: bugs-h...@httpd.apache.org

[Bug 59840] Segmentation fault during the request processing

[bugzilla]

https://bz.apache.org/bugzilla/show_bug.cgi?id=59840

Stefan Eissing  changed:

   What|Removed |Added

 Status|NEW |NEEDINFO

--- Comment #24 from Stefan Eissing  ---
Since the mentioned releases, various bugs have been identified and fixed that
could explain the observed behaviour. Also fuzzing is being done for the HTTP/2
implementation which improved robustness.

If there is no new data to report on a current version, I'd like to close this
ticket.

-- 
You are receiving this mail because:
You are the assignee for the bug.
-
To unsubscribe, e-mail: bugs-unsubscr...@httpd.apache.org
For additional commands, e-mail: bugs-h...@httpd.apache.org

[Bug 59840] Segmentation fault during the request processing

[bugzilla]

https://bz.apache.org/bugzilla/show_bug.cgi?id=59840

cbarb...@okta.com changed:

   What|Removed |Added

 CC||cbarb...@okta.com

-- 
You are receiving this mail because:
You are the assignee for the bug.
-
To unsubscribe, e-mail: bugs-unsubscr...@httpd.apache.org
For additional commands, e-mail: bugs-h...@httpd.apache.org

[Bug 59840] Segmentation fault during the request processing

[bugzilla]

https://bz.apache.org/bugzilla/show_bug.cgi?id=59840

--- Comment #23 from Konstantin J. Chernov  ---
I've looked through the logs: out of 5 crashes we had during the last week,
there were only one crash that corresponds with the warning from mod_http2:
[2016-09-10 14:34:41.832137] [http2:warn] [pid 10635:tid 91] [C:AB0Zx5UDdAA]
AH03198: h2_mplx(172): release, waiting for 87865 seconds now for 1 h2_workers
to return, have still 21 tasks outstanding

But that was not written by the process(PID/TID) that crashed, as the messages
from this pid/tid appears further in the logs.

-- 
You are receiving this mail because:
You are the assignee for the bug.

-
To unsubscribe, e-mail: bugs-unsubscr...@httpd.apache.org
For additional commands, e-mail: bugs-h...@httpd.apache.org

[Bug 59840] Segmentation fault during the request processing

[bugzilla]

https://bz.apache.org/bugzilla/show_bug.cgi?id=59840

--- Comment #22 from Stefan Eissing  ---
Just for completeness since I did not ask yet: do you see anything in the error
logs from mod_http2. Any warning?

-- 
You are receiving this mail because:
You are the assignee for the bug.

-
To unsubscribe, e-mail: bugs-unsubscr...@httpd.apache.org
For additional commands, e-mail: bugs-h...@httpd.apache.org

[Bug 59840] Segmentation fault during the request processing

[bugzilla]

https://bz.apache.org/bugzilla/show_bug.cgi?id=59840

--- Comment #21 from Konstantin J. Chernov  ---
Created attachment 34222
  --> https://bz.apache.org/bugzilla/attachment.cgi?id=34222=edit
three more full pstack outputs

-- 
You are receiving this mail because:
You are the assignee for the bug.

-
To unsubscribe, e-mail: bugs-unsubscr...@httpd.apache.org
For additional commands, e-mail: bugs-h...@httpd.apache.org

[Bug 59840] Segmentation fault during the request processing

[bugzilla]

https://bz.apache.org/bugzilla/show_bug.cgi?id=59840

--- Comment #20 from Eric Covener  ---
(In reply to Stefan Eissing from comment #19)
> Hmm, I do not see it. Someone with better eyes than me?

t94 is suspicious

-- 
You are receiving this mail because:
You are the assignee for the bug.

-
To unsubscribe, e-mail: bugs-unsubscr...@httpd.apache.org
For additional commands, e-mail: bugs-h...@httpd.apache.org

[Bug 59840] Segmentation fault during the request processing

[bugzilla]

https://bz.apache.org/bugzilla/show_bug.cgi?id=59840

--- Comment #19 from Stefan Eissing  ---
Hmm, I do not see it. Someone with better eyes than me?

-- 
You are receiving this mail because:
You are the assignee for the bug.

-
To unsubscribe, e-mail: bugs-unsubscr...@httpd.apache.org
For additional commands, e-mail: bugs-h...@httpd.apache.org

[Bug 59840] Segmentation fault during the request processing

[bugzilla]

https://bz.apache.org/bugzilla/show_bug.cgi?id=59840

--- Comment #17 from Konstantin J. Chernov  ---
Created attachment 34203
  --> https://bz.apache.org/bugzilla/attachment.cgi?id=34203=edit
core.apache.httpd.12673.1473072482.stack

-- 
You are receiving this mail because:
You are the assignee for the bug.

-
To unsubscribe, e-mail: bugs-unsubscr...@httpd.apache.org
For additional commands, e-mail: bugs-h...@httpd.apache.org

[Bug 59840] Segmentation fault during the request processing

[bugzilla]

https://bz.apache.org/bugzilla/show_bug.cgi?id=59840

--- Comment #18 from Konstantin J. Chernov  ---
I've attached full pstack of one of three coredumps:
core.apache.httpd.12673.1473072482.stack
Hopefully it will give more valuable info.

-- 
You are receiving this mail because:
You are the assignee for the bug.

-
To unsubscribe, e-mail: bugs-unsubscr...@httpd.apache.org
For additional commands, e-mail: bugs-h...@httpd.apache.org

[Bug 59840] Segmentation fault during the request processing

[bugzilla]

https://bz.apache.org/bugzilla/show_bug.cgi?id=59840

--- Comment #16 from Eric Covener  ---
(In reply to Konstantin J. Chernov from comment #15)
> After more than a week of stable work, there were 3 segfaults yesterday, all
> with the same stack:
> > ::stack
> libc.so.1`_read+0xa()
> ap_mpm_podx_check+0x38()
> child_main+0x418()
> make_child+0x143()
> perform_idle_server_maintenance+0x67f()
> server_main_loop+0x346()
> event_run+0x320()
> ap_run_mpm+0x71()
> main+0xe89()
> _start+0x6c()
> 
> I'm not really sure they are somehow connected to mod_http2, though without
> mod_http2 there are no segfaults at all.

That is not the crashing thread, there must be some other that looks a little
more suspicious.

-- 
You are receiving this mail because:
You are the assignee for the bug.

-
To unsubscribe, e-mail: bugs-unsubscr...@httpd.apache.org
For additional commands, e-mail: bugs-h...@httpd.apache.org

[Bug 59840] Segmentation fault during the request processing

[bugzilla]

https://bz.apache.org/bugzilla/show_bug.cgi?id=59840

--- Comment #15 from Konstantin J. Chernov  ---
After more than a week of stable work, there were 3 segfaults yesterday, all
with the same stack:
> ::stack
libc.so.1`_read+0xa()
ap_mpm_podx_check+0x38()
child_main+0x418()
make_child+0x143()
perform_idle_server_maintenance+0x67f()
server_main_loop+0x346()
event_run+0x320()
ap_run_mpm+0x71()
main+0xe89()
_start+0x6c()

I'm not really sure they are somehow connected to mod_http2, though without
mod_http2 there are no segfaults at all.

-- 
You are receiving this mail because:
You are the assignee for the bug.

-
To unsubscribe, e-mail: bugs-unsubscr...@httpd.apache.org
For additional commands, e-mail: bugs-h...@httpd.apache.org

[Bug 59840] Segmentation fault during the request processing

[bugzilla]

https://bz.apache.org/bugzilla/show_bug.cgi?id=59840

--- Comment #14 from Konstantin J. Chernov  ---
I've replaced modules/http2 from https://github.com/apache/httpd/tree/2.4.x
(hopefully, this link is correct) and rebuilt the apache with the rest of the
sources left unchanged, then replaced single mod_http2.so on prod environment.

Will see if there are any changes.

-- 
You are receiving this mail because:
You are the assignee for the bug.

-
To unsubscribe, e-mail: bugs-unsubscr...@httpd.apache.org
For additional commands, e-mail: bugs-h...@httpd.apache.org

[Bug 59840] Segmentation fault during the request processing

[bugzilla]

https://bz.apache.org/bugzilla/show_bug.cgi?id=59840

--- Comment #13 from Konstantin J. Chernov  ---
Hi.

Will it be enough to replace mod_http2 folder contents from
https://github.com/apache/httpd/tree/trunk/modules/http2 and recompile the
Apache?

-- 
You are receiving this mail because:
You are the assignee for the bug.

-
To unsubscribe, e-mail: bugs-unsubscr...@httpd.apache.org
For additional commands, e-mail: bugs-h...@httpd.apache.org

[Bug 59840] Segmentation fault during the request processing

[bugzilla]

https://bz.apache.org/bugzilla/show_bug.cgi?id=59840

--- Comment #12 from Stefan Eissing  ---
Konstantin,

are you able to build a new mod_http2 for testing? The current state of branch
2.4.x contains a cleanup fix that might address the issues you observe.

As an alternative, you can also you the github release at
https://github.com/icing/mod_h2/releases/tag/v1.6.1 if you have apxs and
autoconf on your system.

-- 
You are receiving this mail because:
You are the assignee for the bug.

-
To unsubscribe, e-mail: bugs-unsubscr...@httpd.apache.org
For additional commands, e-mail: bugs-h...@httpd.apache.org

[Bug 59840] Segmentation fault during the request processing

[bugzilla]

https://bz.apache.org/bugzilla/show_bug.cgi?id=59840

--- Comment #11 from Stefan Eissing  ---
Thanks Konstantin. At least it's clear that http2 is the culprit and the
backtrace points to connection shutdown. Have do analyze that a bit more...

-- 
You are receiving this mail because:
You are the assignee for the bug.

-
To unsubscribe, e-mail: bugs-unsubscr...@httpd.apache.org
For additional commands, e-mail: bugs-h...@httpd.apache.org

[Bug 59840] Segmentation fault during the request processing

[bugzilla]

https://bz.apache.org/bugzilla/show_bug.cgi?id=59840

--- Comment #10 from Konstantin J. Chernov  ---
Right after enabling http2 again, got a few more segfaults :(

Here's the latest one (+2 more with the same backtraces as in my previous
comments):
> ::stack
libaprutil-1.so.0.5.4`apr_brigade_cleanup+0x43()
libaprutil-1.so.0.5.4`brigade_cleanup+0x18()
libapr-1.so.0.5.2`run_cleanups+0x3c()
libapr-1.so.0.5.2`apr_pool_destroy+0x57()
eor_bucket_destroy+0x2a()
libaprutil-1.so.0.5.4`apr_brigade_cleanup+0x60()
ap_process_request_after_handler+0xc6()
ap_process_async_request+0x79f()
ap_process_request+0x24()
mod_http2.so`h2_task_process_request+0x3fd()
mod_http2.so`h2_task_process_conn+0x228()
ap_run_process_connection+0x61()
mod_http2.so`h2_task_do+0x5e2()
mod_http2.so`execute+0x50()
libapr-1.so.0.5.2`dummy_worker+0x30()
libc.so.1`_thr_setup+0x5b()
libc.so.1`_lwp_start()

-- 
You are receiving this mail because:
You are the assignee for the bug.

-
To unsubscribe, e-mail: bugs-unsubscr...@httpd.apache.org
For additional commands, e-mail: bugs-h...@httpd.apache.org

[Bug 59840] Segmentation fault during the request processing

[bugzilla]

https://bz.apache.org/bugzilla/show_bug.cgi?id=59840

--- Comment #9 from Konstantin J. Chernov  ---
There were no segfaults since 29/07/2016 (the date when I disabled http2
module).
I assume the problem is not caused by 2.4.23 itself :(

-- 
You are receiving this mail because:
You are the assignee for the bug.

-
To unsubscribe, e-mail: bugs-unsubscr...@httpd.apache.org
For additional commands, e-mail: bugs-h...@httpd.apache.org

[Bug 59840] Segmentation fault during the request processing

[bugzilla]

https://bz.apache.org/bugzilla/show_bug.cgi?id=59840

--- Comment #8 from Konstantin J. Chernov  ---
Stefan,

thanks for the reply, I'll disable h2 tomorrow and leave it disabled for about
a week.

-- 
You are receiving this mail because:
You are the assignee for the bug.

-
To unsubscribe, e-mail: bugs-unsubscr...@httpd.apache.org
For additional commands, e-mail: bugs-h...@httpd.apache.org

[Bug 59840] Segmentation fault during the request processing

[bugzilla]

https://bz.apache.org/bugzilla/show_bug.cgi?id=59840

--- Comment #7 from Stefan Eissing  ---
These are all very weird places. I would ask you to disable h2 for the time
being and check if 2.4.23 runs as stable on your platform as 2.4.18 did. Let's
just verify that there not something else here going on.

-- 
You are receiving this mail because:
You are the assignee for the bug.

-
To unsubscribe, e-mail: bugs-unsubscr...@httpd.apache.org
For additional commands, e-mail: bugs-h...@httpd.apache.org

[Bug 59840] Segmentation fault during the request processing

[bugzilla]

https://bz.apache.org/bugzilla/show_bug.cgi?id=59840

--- Comment #6 from Konstantin J. Chernov  ---
Created attachment 34074
  --> https://bz.apache.org/bugzilla/attachment.cgi?id=34074=edit
pstack output of coredumps

-- 
You are receiving this mail because:
You are the assignee for the bug.

-
To unsubscribe, e-mail: bugs-unsubscr...@httpd.apache.org
For additional commands, e-mail: bugs-h...@httpd.apache.org

[Bug 59840] Segmentation fault during the request processing

[bugzilla]

https://bz.apache.org/bugzilla/show_bug.cgi?id=59840

--- Comment #3 from Konstantin J. Chernov  ---
Got another segfault yesterday, though I'm not sure whether it is related to
the original problem:

[2016-07-12 23:58:06.817849] [core:notice] [pid 15804:tid 1] AH00051: child pid
29484 exit signal Segmentation fault (11), possible coredump in
/pub/apache/cores

$ mdb core.apache.httpd.29484.1468346277 
> ::stack
mod_http2.so`h2_brigade_mem_size+0x2e()
mod_http2.so`h2_conn_io_pass+0x33b()
mod_http2.so`on_send_data_cb+0xa45()
libnghttp2.so.14.8.0`nghttp2_session_mem_send_internal+0x165()
libnghttp2.so.14.8.0`nghttp2_session_send+0x59()
mod_http2.so`h2_session_send+0x68()
mod_http2.so`h2_session_process+0x1863()
mod_http2.so`h2_conn_run+0x56()
mod_http2.so`h2_h2_process_conn+0x1122()
ap_run_process_connection+0x61()
process_socket+0x4ab()
worker_thread+0x351()
libapr-1.so.0.5.2`dummy_worker+0x30()
libc.so.1`_thr_setup+0x5b()
libc.so.1`_lwp_start()

-- 
You are receiving this mail because:
You are the assignee for the bug.

-
To unsubscribe, e-mail: bugs-unsubscr...@httpd.apache.org
For additional commands, e-mail: bugs-h...@httpd.apache.org

[Bug 59840] Segmentation fault during the request processing

[bugzilla]

https://bz.apache.org/bugzilla/show_bug.cgi?id=59840

--- Comment #2 from Konstantin J. Chernov  ---
Thanks for the advice!

I removed StdEnvVars from the SSLOptions, will see if there are any changes.

-- 
You are receiving this mail because:
You are the assignee for the bug.

-
To unsubscribe, e-mail: bugs-unsubscr...@httpd.apache.org
For additional commands, e-mail: bugs-h...@httpd.apache.org

[Bug 59840] Segmentation fault during the request processing

[bugzilla]

https://bz.apache.org/bugzilla/show_bug.cgi?id=59840

--- Comment #1 from Stefan Eissing  ---
Thanks for the report. There seem to be issues with concurrent access to SSL
variables when SSLOptions StdEnvVars is active.

I have to check back with the mod_ssl devs on how to best mitigate this issue.
For now, you may try removing the 'StdEnvVars' option - if your application can
live with it - and test if that fixes the issue.

-- 
You are receiving this mail because:
You are the assignee for the bug.

-
To unsubscribe, e-mail: bugs-unsubscr...@httpd.apache.org
For additional commands, e-mail: bugs-h...@httpd.apache.org

[Bug 59840] Segmentation fault during the request processing

[bugzilla]

https://bz.apache.org/bugzilla/show_bug.cgi?id=59840

Michael Kaufmann  changed:

   What|Removed |Added

 CC||apache-bugzilla@michael-kau
   ||fmann.ch

-- 
You are receiving this mail because:
You are the assignee for the bug.

-
To unsubscribe, e-mail: bugs-unsubscr...@httpd.apache.org
For additional commands, e-mail: bugs-h...@httpd.apache.org