Re: hanging apache httpd processes due to mod_h2
looks good so war. Will wait some more days.
Greets,
Stefan
Am 19.04.2017 um 11:09 schrieb Stefan Eissing:
> Thanks, the backtraces really helped. Could you try the following patch?
>
> -Stefan
>
>
>
>
>> Am 19.04.2017 um 08:32 schrieb Stefan Priebe - Profihost AG
>> :
>>
>> Hi Stefan,
>>
>> this is now with v1.10.2 but it does not help.
>>
>> PID old gen:
>>
>> 215608 yes (old gen) 2 no 0 0 0 0 >> 0
>>
>>
>> requests in G state:
>>
>> 2-2 15608 4/36/36 G 252.75 30695 41 22.10.870.87
>> A.B.C.D h2 XYZ:443
>> GET /themes/Frontend/Responsive/frontend/_public/src/js/vendors
>>
>> 2-2 15608 2/5/5 G 69.62 39645 36 104.0 0.180.18
>> A.B.C.D h2 XYZ:443 GET
>> /web/cache/1492526985_9e591f3c4f420ebca36f3c1be30bd5ee.css
>>
>> thread all bt:
>> https://pastebin.com/raw/Cq9ymt9u
>>
>> I think the interesting thread is:
>> Thread 6
>>
>> Greets,
>> Stefan
>>
>> Am 18.04.2017 um 15:06 schrieb Stefan Priebe - Profihost AG:
>>>
>>> Am 18.04.2017 um 15:03 schrieb Stefan Eissing:
Stefan,
that is a 1.10.0, right? That was the first version without nested locking
and I fixed 2 possible dead locks in 1.10.1.
I am about to release a 1.10.2 with added conformity checks and a fix for
client omitting EOF flags. Could you give that one a try?
>>>
>>> Yes sure where is 1.10.2?
>>>
>>> Stefan
>>>
-Stefan
> Am 18.04.2017 um 14:57 schrieb Stefan Priebe - Profihost AG
> :
>
> Hi,
>
> i saw that all of them are still serving one h2 connection.
>
> server-status:
> 0-3 32375 42/64/181 G 30.09 1020776 214 1285.1
> 2.405.81
> h081217236127.dyn.cm.kabsi.at h2 :443GET
> /wp-content/uploads/Bloglr21-8003asd.jpg HTTP/2.0
>
> And they all have Mode of operation => G
>
> thread apply all bt full shows:
> #0 0x7f8e80b687fc in __lll_lock_wait () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> No symbol table info available.
> #1 0x7f8e80b6b6f8 in _L_cond_lock_886 () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> No symbol table info available.
> #2 0x7f8e80b6b464 in __pthread_mutex_cond_lock () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> No symbol table info available.
> #3 0x7f8e80b6650a in pthread_cond_timedwait@@GLIBC_2.3.2 () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> No symbol table info available.
> #4 0x7f8e80d92322 in apr_thread_cond_timedwait
> (cond=0x7f8d7846bc70, mutex=0x7f8d7846bc20, timeout=6000) at
> locks/unix/thread_cond.c:89
> rv =
> then =
> abstime = {tv_sec = 1491843095, tv_nsec = 700198000}
> #5 0x55e62c0951c5 in r_wait_space (premain=,
> pbl=0x7f8de853da60, block=APR_BLOCK_READ, beam=0x7f8d7846bb08) at
> h2_bucket_beam.c:337
> status =
> #6 append_bucket (pbl=0x7f8de853da60, block=APR_BLOCK_READ,
> b=0x7f8d7846dd38, beam=0x7f8d7846bb08) at h2_bucket_beam.c:776
> data = 0x7f8de853dac8 "\240\037"
> len = 0
> space_left = 0
> status =
> #7 h2_beam_send (beam=0x7f8d7846bb08, sender_bb=0x7f8d7864ba90,
> block=APR_BLOCK_READ) at h2_bucket_beam.c:906
> force_report = 1
> b = 0x7f8d7846dd38
> status = 0
> bl = {mutex = 0x7f8d7846bc20, leave = 0x55e62c094250
> , leave_ctx = 0x0}
> #8 0x55e62c08f3aa in send_out (task=0x7f8d7846f740,
> bb=0x7f8d7864ba90, block=1) at h2_task.c:100
> written = 8096
> left = 140245585033024
> status = 1
> #9 0x55e62c08f976 in slave_out (task=0x7f8d7846f740,
> f=0x7f8d7846bdd8, bb=0x7f8d7864ba90) at h2_task.c:177
> status = -512
> flush = 0
> blocking = 1
> #10 0x55e62c08fbfd in h2_filter_slave_output (filter=0x7f8d7846bdd8,
> brigade=0x7f8d7864ba90) at h2_task.c:370
> task = 0x7f8d7846f740
> status =
> #11 0x55e62bffc01c in ap_content_length_filter (f=0x7f8d78472ca8,
> b=0x7f8d7864ba90) at protocol.c:1713
> r = 0x7f8d78471750
> ctx = 0x7f8d7864bc90
> e = 0x7f8d7864ba98
> eblock = (unknown: 2159446012)
> #12 0x55e62c04853d in deflate_out_filter (f=0x7f8d7864b650,
> bb=0x7f8d7864b8f8) at mod_deflate.c:961
> rv = -512
> r = 0x7f8d78471750
> ctx = 0x7f8d7864b9a8
> len = 8096
> blen = 1157663
> data = 0x7f8d7e40b000 "/*! jQuery v2.1.4 | (c) 2005, 2015 jQuery
> Foundation, Inc. | jquery.org/license
> */\n!function(a,b){\"object\"==typeof module&&\"object\"==typeof
> module.exports?module.exports=a.document?b(a,!0):function(a)"...
> c = 0x55e62c7a7498
> #13 0x55e62c044b1d in filter_harness (f=0x7f8d
Re: hanging apache httpd processes due to mod_h2
Thanks, the backtraces really helped. Could you try the following patch?
-Stefan
h2_beam_locks_v1.diff
Description: Binary data
> Am 19.04.2017 um 08:32 schrieb Stefan Priebe - Profihost AG
> :
>
> Hi Stefan,
>
> this is now with v1.10.2 but it does not help.
>
> PID old gen:
>
> 2 15608 yes (old gen) 2 no 0 0 0 0 > 0
>
>
> requests in G state:
>
> 2-2 15608 4/36/36 G 252.75 30695 41 22.10.870.87
> A.B.C.D h2 XYZ:443
> GET /themes/Frontend/Responsive/frontend/_public/src/js/vendors
>
> 2-2 15608 2/5/5 G 69.62 39645 36 104.0 0.180.18
> A.B.C.D h2 XYZ:443 GET
> /web/cache/1492526985_9e591f3c4f420ebca36f3c1be30bd5ee.css
>
> thread all bt:
> https://pastebin.com/raw/Cq9ymt9u
>
> I think the interesting thread is:
> Thread 6
>
> Greets,
> Stefan
>
> Am 18.04.2017 um 15:06 schrieb Stefan Priebe - Profihost AG:
>>
>> Am 18.04.2017 um 15:03 schrieb Stefan Eissing:
>>> Stefan,
>>>
>>> that is a 1.10.0, right? That was the first version without nested locking
>>> and I fixed 2 possible dead locks in 1.10.1.
>>>
>>> I am about to release a 1.10.2 with added conformity checks and a fix for
>>> client omitting EOF flags. Could you give that one a try?
>>
>> Yes sure where is 1.10.2?
>>
>> Stefan
>>
>>>
>>> -Stefan
>>>
Am 18.04.2017 um 14:57 schrieb Stefan Priebe - Profihost AG
:
Hi,
i saw that all of them are still serving one h2 connection.
server-status:
0-332375 42/64/181 G 30.09 1020776 214 1285.1
2.405.81
h081217236127.dyn.cm.kabsi.at h2 :443GET
/wp-content/uploads/Bloglr21-8003asd.jpg HTTP/2.0
And they all have Mode of operation => G
thread apply all bt full shows:
#0 0x7f8e80b687fc in __lll_lock_wait () from
/lib/x86_64-linux-gnu/libpthread.so.0
No symbol table info available.
#1 0x7f8e80b6b6f8 in _L_cond_lock_886 () from
/lib/x86_64-linux-gnu/libpthread.so.0
No symbol table info available.
#2 0x7f8e80b6b464 in __pthread_mutex_cond_lock () from
/lib/x86_64-linux-gnu/libpthread.so.0
No symbol table info available.
#3 0x7f8e80b6650a in pthread_cond_timedwait@@GLIBC_2.3.2 () from
/lib/x86_64-linux-gnu/libpthread.so.0
No symbol table info available.
#4 0x7f8e80d92322 in apr_thread_cond_timedwait
(cond=0x7f8d7846bc70, mutex=0x7f8d7846bc20, timeout=6000) at
locks/unix/thread_cond.c:89
rv =
then =
abstime = {tv_sec = 1491843095, tv_nsec = 700198000}
#5 0x55e62c0951c5 in r_wait_space (premain=,
pbl=0x7f8de853da60, block=APR_BLOCK_READ, beam=0x7f8d7846bb08) at
h2_bucket_beam.c:337
status =
#6 append_bucket (pbl=0x7f8de853da60, block=APR_BLOCK_READ,
b=0x7f8d7846dd38, beam=0x7f8d7846bb08) at h2_bucket_beam.c:776
data = 0x7f8de853dac8 "\240\037"
len = 0
space_left = 0
status =
#7 h2_beam_send (beam=0x7f8d7846bb08, sender_bb=0x7f8d7864ba90,
block=APR_BLOCK_READ) at h2_bucket_beam.c:906
force_report = 1
b = 0x7f8d7846dd38
status = 0
bl = {mutex = 0x7f8d7846bc20, leave = 0x55e62c094250
, leave_ctx = 0x0}
#8 0x55e62c08f3aa in send_out (task=0x7f8d7846f740,
bb=0x7f8d7864ba90, block=1) at h2_task.c:100
written = 8096
left = 140245585033024
status = 1
#9 0x55e62c08f976 in slave_out (task=0x7f8d7846f740,
f=0x7f8d7846bdd8, bb=0x7f8d7864ba90) at h2_task.c:177
status = -512
flush = 0
blocking = 1
#10 0x55e62c08fbfd in h2_filter_slave_output (filter=0x7f8d7846bdd8,
brigade=0x7f8d7864ba90) at h2_task.c:370
task = 0x7f8d7846f740
status =
#11 0x55e62bffc01c in ap_content_length_filter (f=0x7f8d78472ca8,
b=0x7f8d7864ba90) at protocol.c:1713
r = 0x7f8d78471750
ctx = 0x7f8d7864bc90
e = 0x7f8d7864ba98
eblock = (unknown: 2159446012)
#12 0x55e62c04853d in deflate_out_filter (f=0x7f8d7864b650,
bb=0x7f8d7864b8f8) at mod_deflate.c:961
rv = -512
r = 0x7f8d78471750
ctx = 0x7f8d7864b9a8
len = 8096
blen = 1157663
data = 0x7f8d7e40b000 "/*! jQuery v2.1.4 | (c) 2005, 2015 jQuery
Foundation, Inc. | jquery.org/license
*/\n!function(a,b){\"object\"==typeof module&&\"object\"==typeof
module.exports?module.exports=a.document?b(a,!0):function(a)"...
c = 0x55e62c7a7498
#13 0x55e62c044b1d in filter_harness (f=0x7f8d7846bc28, bb=0x80) at
mod_filter.c:323
ret = -512
cachecontrol = 0xfe00 >>> at address 0xfe00>
filter = 0xe
#14 0
Re: hanging apache httpd processes due to mod_h2
Hi Stefan,
this is now with v1.10.2 but it does not help.
PID old gen:
2 15608 yes (old gen) 2 no 0 0 0 0 0
requests in G state:
2-2 15608 4/36/36 G 252.75 30695 41 22.10.870.87
A.B.C.D h2 XYZ:443
GET /themes/Frontend/Responsive/frontend/_public/src/js/vendors
2-2 15608 2/5/5 G 69.62 39645 36 104.0 0.180.18
A.B.C.D h2 XYZ:443 GET
/web/cache/1492526985_9e591f3c4f420ebca36f3c1be30bd5ee.css
thread all bt:
https://pastebin.com/raw/Cq9ymt9u
I think the interesting thread is:
Thread 6
Greets,
Stefan
Am 18.04.2017 um 15:06 schrieb Stefan Priebe - Profihost AG:
>
> Am 18.04.2017 um 15:03 schrieb Stefan Eissing:
>> Stefan,
>>
>> that is a 1.10.0, right? That was the first version without nested locking
>> and I fixed 2 possible dead locks in 1.10.1.
>>
>> I am about to release a 1.10.2 with added conformity checks and a fix for
>> client omitting EOF flags. Could you give that one a try?
>
> Yes sure where is 1.10.2?
>
> Stefan
>
>>
>> -Stefan
>>
>>> Am 18.04.2017 um 14:57 schrieb Stefan Priebe - Profihost AG
>>> :
>>>
>>> Hi,
>>>
>>> i saw that all of them are still serving one h2 connection.
>>>
>>> server-status:
>>> 0-3 32375 42/64/181 G 30.09 1020776 214 1285.1 2.40
>>> 5.81
>>> h081217236127.dyn.cm.kabsi.at h2 :443GET
>>> /wp-content/uploads/Bloglr21-8003asd.jpg HTTP/2.0
>>>
>>> And they all have Mode of operation => G
>>>
>>> thread apply all bt full shows:
>>> #0 0x7f8e80b687fc in __lll_lock_wait () from
>>> /lib/x86_64-linux-gnu/libpthread.so.0
>>> No symbol table info available.
>>> #1 0x7f8e80b6b6f8 in _L_cond_lock_886 () from
>>> /lib/x86_64-linux-gnu/libpthread.so.0
>>> No symbol table info available.
>>> #2 0x7f8e80b6b464 in __pthread_mutex_cond_lock () from
>>> /lib/x86_64-linux-gnu/libpthread.so.0
>>> No symbol table info available.
>>> #3 0x7f8e80b6650a in pthread_cond_timedwait@@GLIBC_2.3.2 () from
>>> /lib/x86_64-linux-gnu/libpthread.so.0
>>> No symbol table info available.
>>> #4 0x7f8e80d92322 in apr_thread_cond_timedwait
>>> (cond=0x7f8d7846bc70, mutex=0x7f8d7846bc20, timeout=6000) at
>>> locks/unix/thread_cond.c:89
>>>rv =
>>>then =
>>>abstime = {tv_sec = 1491843095, tv_nsec = 700198000}
>>> #5 0x55e62c0951c5 in r_wait_space (premain=,
>>> pbl=0x7f8de853da60, block=APR_BLOCK_READ, beam=0x7f8d7846bb08) at
>>> h2_bucket_beam.c:337
>>>status =
>>> #6 append_bucket (pbl=0x7f8de853da60, block=APR_BLOCK_READ,
>>> b=0x7f8d7846dd38, beam=0x7f8d7846bb08) at h2_bucket_beam.c:776
>>>data = 0x7f8de853dac8 "\240\037"
>>>len = 0
>>>space_left = 0
>>>status =
>>> #7 h2_beam_send (beam=0x7f8d7846bb08, sender_bb=0x7f8d7864ba90,
>>> block=APR_BLOCK_READ) at h2_bucket_beam.c:906
>>>force_report = 1
>>>b = 0x7f8d7846dd38
>>>status = 0
>>>bl = {mutex = 0x7f8d7846bc20, leave = 0x55e62c094250
>>> , leave_ctx = 0x0}
>>> #8 0x55e62c08f3aa in send_out (task=0x7f8d7846f740,
>>> bb=0x7f8d7864ba90, block=1) at h2_task.c:100
>>>written = 8096
>>>left = 140245585033024
>>>status = 1
>>> #9 0x55e62c08f976 in slave_out (task=0x7f8d7846f740,
>>> f=0x7f8d7846bdd8, bb=0x7f8d7864ba90) at h2_task.c:177
>>>status = -512
>>>flush = 0
>>>blocking = 1
>>> #10 0x55e62c08fbfd in h2_filter_slave_output (filter=0x7f8d7846bdd8,
>>> brigade=0x7f8d7864ba90) at h2_task.c:370
>>>task = 0x7f8d7846f740
>>>status =
>>> #11 0x55e62bffc01c in ap_content_length_filter (f=0x7f8d78472ca8,
>>> b=0x7f8d7864ba90) at protocol.c:1713
>>>r = 0x7f8d78471750
>>>ctx = 0x7f8d7864bc90
>>>e = 0x7f8d7864ba98
>>>eblock = (unknown: 2159446012)
>>> #12 0x55e62c04853d in deflate_out_filter (f=0x7f8d7864b650,
>>> bb=0x7f8d7864b8f8) at mod_deflate.c:961
>>>rv = -512
>>>r = 0x7f8d78471750
>>>ctx = 0x7f8d7864b9a8
>>>len = 8096
>>>blen = 1157663
>>>data = 0x7f8d7e40b000 "/*! jQuery v2.1.4 | (c) 2005, 2015 jQuery
>>> Foundation, Inc. | jquery.org/license
>>> */\n!function(a,b){\"object\"==typeof module&&\"object\"==typeof
>>> module.exports?module.exports=a.document?b(a,!0):function(a)"...
>>>c = 0x55e62c7a7498
>>> #13 0x55e62c044b1d in filter_harness (f=0x7f8d7846bc28, bb=0x80) at
>>> mod_filter.c:323
>>>ret = -512
>>>cachecontrol = 0xfe00 >> at address 0xfe00>
>>>filter = 0xe
>>> #14 0x55e62c002242 in default_handler (r=0x7f8d78471750) at core.c:4746
>>>c = 0x7f8d7846b720
>>>bb = 0x7f8d7864b8f8
>>>e = 0xfe00
>>>d = 0x7f8d7864dfa0
>>>fd = 0x7f8d7864b7b8
>>>bld_content_md5 = 2019866872
>>> #15 0x55e62c014130 in ap_run_handler (r=r@entry=0x7f8d78471750) at
>>>
