Re: hanging apache httpd processes due to mod_h2
looks good so war. Will wait some more days.
Greets,
Stefan
Am 19.04.2017 um 11:09 schrieb Stefan Eissing:
> Thanks, the backtraces really helped. Could you try the following patch?
>
> -Stefan
>
>
>
>
>> Am 19.04.2017 um 08:32 schrieb Stefan Priebe - Profihost AG
>> :
>>
>> Hi Stefan,
>>
>> this is now with v1.10.2 but it does not help.
>>
>> PID old gen:
>>
>> 215608 yes (old gen) 2 no 0 0 0 0 >> 0
>>
>>
>> requests in G state:
>>
>> 2-2 15608 4/36/36 G 252.75 30695 41 22.10.870.87
>> A.B.C.D h2 XYZ:443
>> GET /themes/Frontend/Responsive/frontend/_public/src/js/vendors
>>
>> 2-2 15608 2/5/5 G 69.62 39645 36 104.0 0.180.18
>> A.B.C.D h2 XYZ:443 GET
>> /web/cache/1492526985_9e591f3c4f420ebca36f3c1be30bd5ee.css
>>
>> thread all bt:
>> https://pastebin.com/raw/Cq9ymt9u
>>
>> I think the interesting thread is:
>> Thread 6
>>
>> Greets,
>> Stefan
>>
>> Am 18.04.2017 um 15:06 schrieb Stefan Priebe - Profihost AG:
>>>
>>> Am 18.04.2017 um 15:03 schrieb Stefan Eissing:
Stefan,
that is a 1.10.0, right? That was the first version without nested locking
and I fixed 2 possible dead locks in 1.10.1.
I am about to release a 1.10.2 with added conformity checks and a fix for
client omitting EOF flags. Could you give that one a try?
>>>
>>> Yes sure where is 1.10.2?
>>>
>>> Stefan
>>>
-Stefan
> Am 18.04.2017 um 14:57 schrieb Stefan Priebe - Profihost AG
> :
>
> Hi,
>
> i saw that all of them are still serving one h2 connection.
>
> server-status:
> 0-3 32375 42/64/181 G 30.09 1020776 214 1285.1
> 2.405.81
> h081217236127.dyn.cm.kabsi.at h2 :443GET
> /wp-content/uploads/Bloglr21-8003asd.jpg HTTP/2.0
>
> And they all have Mode of operation => G
>
> thread apply all bt full shows:
> #0 0x7f8e80b687fc in __lll_lock_wait () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> No symbol table info available.
> #1 0x7f8e80b6b6f8 in _L_cond_lock_886 () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> No symbol table info available.
> #2 0x7f8e80b6b464 in __pthread_mutex_cond_lock () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> No symbol table info available.
> #3 0x7f8e80b6650a in pthread_cond_timedwait@@GLIBC_2.3.2 () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> No symbol table info available.
> #4 0x7f8e80d92322 in apr_thread_cond_timedwait
> (cond=0x7f8d7846bc70, mutex=0x7f8d7846bc20, timeout=6000) at
> locks/unix/thread_cond.c:89
> rv =
> then =
> abstime = {tv_sec = 1491843095, tv_nsec = 700198000}
> #5 0x55e62c0951c5 in r_wait_space (premain=,
> pbl=0x7f8de853da60, block=APR_BLOCK_READ, beam=0x7f8d7846bb08) at
> h2_bucket_beam.c:337
> status =
> #6 append_bucket (pbl=0x7f8de853da60, block=APR_BLOCK_READ,
> b=0x7f8d7846dd38, beam=0x7f8d7846bb08) at h2_bucket_beam.c:776
> data = 0x7f8de853dac8 "\240\037"
> len = 0
> space_left = 0
> status =
> #7 h2_beam_send (beam=0x7f8d7846bb08, sender_bb=0x7f8d7864ba90,
> block=APR_BLOCK_READ) at h2_bucket_beam.c:906
> force_report = 1
> b = 0x7f8d7846dd38
> status = 0
> bl = {mutex = 0x7f8d7846bc20, leave = 0x55e62c094250
> , leave_ctx = 0x0}
> #8 0x55e62c08f3aa in send_out (task=0x7f8d7846f740,
> bb=0x7f8d7864ba90, block=1) at h2_task.c:100
> written = 8096
> left = 140245585033024
> status = 1
> #9 0x55e62c08f976 in slave_out (task=0x7f8d7846f740,
> f=0x7f8d7846bdd8, bb=0x7f8d7864ba90) at h2_task.c:177
> status = -512
> flush = 0
> blocking = 1
> #10 0x55e62c08fbfd in h2_filter_slave_output (filter=0x7f8d7846bdd8,
> brigade=0x7f8d7864ba90) at h2_task.c:370
> task = 0x7f8d7846f740
> status =
> #11 0x55e62bffc01c in ap_content_length_filter (f=0x7f8d78472ca8,
> b=0x7f8d7864ba90) at protocol.c:1713
> r = 0x7f8d78471750
> ctx = 0x7f8d7864bc90
> e = 0x7f8d7864ba98
> eblock = (unknown: 2159446012)
> #12 0x55e62c04853d in deflate_out_filter (f=0x7f8d7864b650,
> bb=0x7f8d7864b8f8) at mod_deflate.c:961
> rv = -512
> r = 0x7f8d78471750
> ctx = 0x7f8d7864b9a8
> len = 8096
> blen = 1157663
> data = 0x7f8d7e40b000 "/*! jQuery v2.1.4 | (c) 2005, 2015 jQuery
> Foundation, Inc. | jquery.org/license
> */\n!function(a,b){\"object\"==typeof module&&\"object\"==typeof
> module.exports?module.exports=a.document?b(a,!0):function(a)"...
> c = 0x55e62c7a7498
> #13 0x55e62c044b1d in filter_harness (f=0x7f8d
Re: hanging apache httpd processes due to mod_h2
Thanks, the backtraces really helped. Could you try the following patch?
-Stefan
h2_beam_locks_v1.diff
Description: Binary data
> Am 19.04.2017 um 08:32 schrieb Stefan Priebe - Profihost AG
> :
>
> Hi Stefan,
>
> this is now with v1.10.2 but it does not help.
>
> PID old gen:
>
> 2 15608 yes (old gen) 2 no 0 0 0 0 > 0
>
>
> requests in G state:
>
> 2-2 15608 4/36/36 G 252.75 30695 41 22.10.870.87
> A.B.C.D h2 XYZ:443
> GET /themes/Frontend/Responsive/frontend/_public/src/js/vendors
>
> 2-2 15608 2/5/5 G 69.62 39645 36 104.0 0.180.18
> A.B.C.D h2 XYZ:443 GET
> /web/cache/1492526985_9e591f3c4f420ebca36f3c1be30bd5ee.css
>
> thread all bt:
> https://pastebin.com/raw/Cq9ymt9u
>
> I think the interesting thread is:
> Thread 6
>
> Greets,
> Stefan
>
> Am 18.04.2017 um 15:06 schrieb Stefan Priebe - Profihost AG:
>>
>> Am 18.04.2017 um 15:03 schrieb Stefan Eissing:
>>> Stefan,
>>>
>>> that is a 1.10.0, right? That was the first version without nested locking
>>> and I fixed 2 possible dead locks in 1.10.1.
>>>
>>> I am about to release a 1.10.2 with added conformity checks and a fix for
>>> client omitting EOF flags. Could you give that one a try?
>>
>> Yes sure where is 1.10.2?
>>
>> Stefan
>>
>>>
>>> -Stefan
>>>
Am 18.04.2017 um 14:57 schrieb Stefan Priebe - Profihost AG
:
Hi,
i saw that all of them are still serving one h2 connection.
server-status:
0-332375 42/64/181 G 30.09 1020776 214 1285.1
2.405.81
h081217236127.dyn.cm.kabsi.at h2 :443GET
/wp-content/uploads/Bloglr21-8003asd.jpg HTTP/2.0
And they all have Mode of operation => G
thread apply all bt full shows:
#0 0x7f8e80b687fc in __lll_lock_wait () from
/lib/x86_64-linux-gnu/libpthread.so.0
No symbol table info available.
#1 0x7f8e80b6b6f8 in _L_cond_lock_886 () from
/lib/x86_64-linux-gnu/libpthread.so.0
No symbol table info available.
#2 0x7f8e80b6b464 in __pthread_mutex_cond_lock () from
/lib/x86_64-linux-gnu/libpthread.so.0
No symbol table info available.
#3 0x7f8e80b6650a in pthread_cond_timedwait@@GLIBC_2.3.2 () from
/lib/x86_64-linux-gnu/libpthread.so.0
No symbol table info available.
#4 0x7f8e80d92322 in apr_thread_cond_timedwait
(cond=0x7f8d7846bc70, mutex=0x7f8d7846bc20, timeout=6000) at
locks/unix/thread_cond.c:89
rv =
then =
abstime = {tv_sec = 1491843095, tv_nsec = 700198000}
#5 0x55e62c0951c5 in r_wait_space (premain=,
pbl=0x7f8de853da60, block=APR_BLOCK_READ, beam=0x7f8d7846bb08) at
h2_bucket_beam.c:337
status =
#6 append_bucket (pbl=0x7f8de853da60, block=APR_BLOCK_READ,
b=0x7f8d7846dd38, beam=0x7f8d7846bb08) at h2_bucket_beam.c:776
data = 0x7f8de853dac8 "\240\037"
len = 0
space_left = 0
status =
#7 h2_beam_send (beam=0x7f8d7846bb08, sender_bb=0x7f8d7864ba90,
block=APR_BLOCK_READ) at h2_bucket_beam.c:906
force_report = 1
b = 0x7f8d7846dd38
status = 0
bl = {mutex = 0x7f8d7846bc20, leave = 0x55e62c094250
, leave_ctx = 0x0}
#8 0x55e62c08f3aa in send_out (task=0x7f8d7846f740,
bb=0x7f8d7864ba90, block=1) at h2_task.c:100
written = 8096
left = 140245585033024
status = 1
#9 0x55e62c08f976 in slave_out (task=0x7f8d7846f740,
f=0x7f8d7846bdd8, bb=0x7f8d7864ba90) at h2_task.c:177
status = -512
flush = 0
blocking = 1
#10 0x55e62c08fbfd in h2_filter_slave_output (filter=0x7f8d7846bdd8,
brigade=0x7f8d7864ba90) at h2_task.c:370
task = 0x7f8d7846f740
status =
#11 0x55e62bffc01c in ap_content_length_filter (f=0x7f8d78472ca8,
b=0x7f8d7864ba90) at protocol.c:1713
r = 0x7f8d78471750
ctx = 0x7f8d7864bc90
e = 0x7f8d7864ba98
eblock = (unknown: 2159446012)
#12 0x55e62c04853d in deflate_out_filter (f=0x7f8d7864b650,
bb=0x7f8d7864b8f8) at mod_deflate.c:961
rv = -512
r = 0x7f8d78471750
ctx = 0x7f8d7864b9a8
len = 8096
blen = 1157663
data = 0x7f8d7e40b000 "/*! jQuery v2.1.4 | (c) 2005, 2015 jQuery
Foundation, Inc. | jquery.org/license
*/\n!function(a,b){\"object\"==typeof module&&\"object\"==typeof
module.exports?module.exports=a.document?b(a,!0):function(a)"...
c = 0x55e62c7a7498
#13 0x55e62c044b1d in filter_harness (f=0x7f8d7846bc28, bb=0x80) at
mod_filter.c:323
ret = -512
cachecontrol = 0xfe00 >>> at address 0xfe00>
filter = 0xe
#14 0
Re: hanging apache httpd processes due to mod_h2
Hi Stefan,
this is now with v1.10.2 but it does not help.
PID old gen:
2 15608 yes (old gen) 2 no 0 0 0 0 0
requests in G state:
2-2 15608 4/36/36 G 252.75 30695 41 22.10.870.87
A.B.C.D h2 XYZ:443
GET /themes/Frontend/Responsive/frontend/_public/src/js/vendors
2-2 15608 2/5/5 G 69.62 39645 36 104.0 0.180.18
A.B.C.D h2 XYZ:443 GET
/web/cache/1492526985_9e591f3c4f420ebca36f3c1be30bd5ee.css
thread all bt:
https://pastebin.com/raw/Cq9ymt9u
I think the interesting thread is:
Thread 6
Greets,
Stefan
Am 18.04.2017 um 15:06 schrieb Stefan Priebe - Profihost AG:
>
> Am 18.04.2017 um 15:03 schrieb Stefan Eissing:
>> Stefan,
>>
>> that is a 1.10.0, right? That was the first version without nested locking
>> and I fixed 2 possible dead locks in 1.10.1.
>>
>> I am about to release a 1.10.2 with added conformity checks and a fix for
>> client omitting EOF flags. Could you give that one a try?
>
> Yes sure where is 1.10.2?
>
> Stefan
>
>>
>> -Stefan
>>
>>> Am 18.04.2017 um 14:57 schrieb Stefan Priebe - Profihost AG
>>> :
>>>
>>> Hi,
>>>
>>> i saw that all of them are still serving one h2 connection.
>>>
>>> server-status:
>>> 0-3 32375 42/64/181 G 30.09 1020776 214 1285.1 2.40
>>> 5.81
>>> h081217236127.dyn.cm.kabsi.at h2 :443GET
>>> /wp-content/uploads/Bloglr21-8003asd.jpg HTTP/2.0
>>>
>>> And they all have Mode of operation => G
>>>
>>> thread apply all bt full shows:
>>> #0 0x7f8e80b687fc in __lll_lock_wait () from
>>> /lib/x86_64-linux-gnu/libpthread.so.0
>>> No symbol table info available.
>>> #1 0x7f8e80b6b6f8 in _L_cond_lock_886 () from
>>> /lib/x86_64-linux-gnu/libpthread.so.0
>>> No symbol table info available.
>>> #2 0x7f8e80b6b464 in __pthread_mutex_cond_lock () from
>>> /lib/x86_64-linux-gnu/libpthread.so.0
>>> No symbol table info available.
>>> #3 0x7f8e80b6650a in pthread_cond_timedwait@@GLIBC_2.3.2 () from
>>> /lib/x86_64-linux-gnu/libpthread.so.0
>>> No symbol table info available.
>>> #4 0x7f8e80d92322 in apr_thread_cond_timedwait
>>> (cond=0x7f8d7846bc70, mutex=0x7f8d7846bc20, timeout=6000) at
>>> locks/unix/thread_cond.c:89
>>>rv =
>>>then =
>>>abstime = {tv_sec = 1491843095, tv_nsec = 700198000}
>>> #5 0x55e62c0951c5 in r_wait_space (premain=,
>>> pbl=0x7f8de853da60, block=APR_BLOCK_READ, beam=0x7f8d7846bb08) at
>>> h2_bucket_beam.c:337
>>>status =
>>> #6 append_bucket (pbl=0x7f8de853da60, block=APR_BLOCK_READ,
>>> b=0x7f8d7846dd38, beam=0x7f8d7846bb08) at h2_bucket_beam.c:776
>>>data = 0x7f8de853dac8 "\240\037"
>>>len = 0
>>>space_left = 0
>>>status =
>>> #7 h2_beam_send (beam=0x7f8d7846bb08, sender_bb=0x7f8d7864ba90,
>>> block=APR_BLOCK_READ) at h2_bucket_beam.c:906
>>>force_report = 1
>>>b = 0x7f8d7846dd38
>>>status = 0
>>>bl = {mutex = 0x7f8d7846bc20, leave = 0x55e62c094250
>>> , leave_ctx = 0x0}
>>> #8 0x55e62c08f3aa in send_out (task=0x7f8d7846f740,
>>> bb=0x7f8d7864ba90, block=1) at h2_task.c:100
>>>written = 8096
>>>left = 140245585033024
>>>status = 1
>>> #9 0x55e62c08f976 in slave_out (task=0x7f8d7846f740,
>>> f=0x7f8d7846bdd8, bb=0x7f8d7864ba90) at h2_task.c:177
>>>status = -512
>>>flush = 0
>>>blocking = 1
>>> #10 0x55e62c08fbfd in h2_filter_slave_output (filter=0x7f8d7846bdd8,
>>> brigade=0x7f8d7864ba90) at h2_task.c:370
>>>task = 0x7f8d7846f740
>>>status =
>>> #11 0x55e62bffc01c in ap_content_length_filter (f=0x7f8d78472ca8,
>>> b=0x7f8d7864ba90) at protocol.c:1713
>>>r = 0x7f8d78471750
>>>ctx = 0x7f8d7864bc90
>>>e = 0x7f8d7864ba98
>>>eblock = (unknown: 2159446012)
>>> #12 0x55e62c04853d in deflate_out_filter (f=0x7f8d7864b650,
>>> bb=0x7f8d7864b8f8) at mod_deflate.c:961
>>>rv = -512
>>>r = 0x7f8d78471750
>>>ctx = 0x7f8d7864b9a8
>>>len = 8096
>>>blen = 1157663
>>>data = 0x7f8d7e40b000 "/*! jQuery v2.1.4 | (c) 2005, 2015 jQuery
>>> Foundation, Inc. | jquery.org/license
>>> */\n!function(a,b){\"object\"==typeof module&&\"object\"==typeof
>>> module.exports?module.exports=a.document?b(a,!0):function(a)"...
>>>c = 0x55e62c7a7498
>>> #13 0x55e62c044b1d in filter_harness (f=0x7f8d7846bc28, bb=0x80) at
>>> mod_filter.c:323
>>>ret = -512
>>>cachecontrol = 0xfe00 >> at address 0xfe00>
>>>filter = 0xe
>>> #14 0x55e62c002242 in default_handler (r=0x7f8d78471750) at core.c:4746
>>>c = 0x7f8d7846b720
>>>bb = 0x7f8d7864b8f8
>>>e = 0xfe00
>>>d = 0x7f8d7864dfa0
>>>fd = 0x7f8d7864b7b8
>>>bld_content_md5 = 2019866872
>>> #15 0x55e62c014130 in ap_run_handler (r=r@entry=0x7f8d78471750) at
>>>
Re: hanging apache httpd processes
Here we go: https://github.com/icing/mod_h2/releases/tag/v1.10.2
> Am 18.04.2017 um 15:11 schrieb Stefan Eissing :
>
> In transit. Just some minutes away...
>
>> Am 18.04.2017 um 15:06 schrieb Stefan Priebe - Profihost AG
>> :
>>
>>
>> Am 18.04.2017 um 15:03 schrieb Stefan Eissing:
>>> Stefan,
>>>
>>> that is a 1.10.0, right? That was the first version without nested locking
>>> and I fixed 2 possible dead locks in 1.10.1.
>>>
>>> I am about to release a 1.10.2 with added conformity checks and a fix for
>>> client omitting EOF flags. Could you give that one a try?
>>
>> Yes sure where is 1.10.2?
>>
>> Stefan
>>
>>>
>>> -Stefan
>>>
Am 18.04.2017 um 14:57 schrieb Stefan Priebe - Profihost AG
:
Hi,
i saw that all of them are still serving one h2 connection.
server-status:
0-332375 42/64/181 G 30.09 1020776 214 1285.1
2.405.81
h081217236127.dyn.cm.kabsi.at h2 :443GET
/wp-content/uploads/Bloglr21-8003asd.jpg HTTP/2.0
And they all have Mode of operation => G
thread apply all bt full shows:
#0 0x7f8e80b687fc in __lll_lock_wait () from
/lib/x86_64-linux-gnu/libpthread.so.0
No symbol table info available.
#1 0x7f8e80b6b6f8 in _L_cond_lock_886 () from
/lib/x86_64-linux-gnu/libpthread.so.0
No symbol table info available.
#2 0x7f8e80b6b464 in __pthread_mutex_cond_lock () from
/lib/x86_64-linux-gnu/libpthread.so.0
No symbol table info available.
#3 0x7f8e80b6650a in pthread_cond_timedwait@@GLIBC_2.3.2 () from
/lib/x86_64-linux-gnu/libpthread.so.0
No symbol table info available.
#4 0x7f8e80d92322 in apr_thread_cond_timedwait
(cond=0x7f8d7846bc70, mutex=0x7f8d7846bc20, timeout=6000) at
locks/unix/thread_cond.c:89
rv =
then =
abstime = {tv_sec = 1491843095, tv_nsec = 700198000}
#5 0x55e62c0951c5 in r_wait_space (premain=,
pbl=0x7f8de853da60, block=APR_BLOCK_READ, beam=0x7f8d7846bb08) at
h2_bucket_beam.c:337
status =
#6 append_bucket (pbl=0x7f8de853da60, block=APR_BLOCK_READ,
b=0x7f8d7846dd38, beam=0x7f8d7846bb08) at h2_bucket_beam.c:776
data = 0x7f8de853dac8 "\240\037"
len = 0
space_left = 0
status =
#7 h2_beam_send (beam=0x7f8d7846bb08, sender_bb=0x7f8d7864ba90,
block=APR_BLOCK_READ) at h2_bucket_beam.c:906
force_report = 1
b = 0x7f8d7846dd38
status = 0
bl = {mutex = 0x7f8d7846bc20, leave = 0x55e62c094250
, leave_ctx = 0x0}
#8 0x55e62c08f3aa in send_out (task=0x7f8d7846f740,
bb=0x7f8d7864ba90, block=1) at h2_task.c:100
written = 8096
left = 140245585033024
status = 1
#9 0x55e62c08f976 in slave_out (task=0x7f8d7846f740,
f=0x7f8d7846bdd8, bb=0x7f8d7864ba90) at h2_task.c:177
status = -512
flush = 0
blocking = 1
#10 0x55e62c08fbfd in h2_filter_slave_output (filter=0x7f8d7846bdd8,
brigade=0x7f8d7864ba90) at h2_task.c:370
task = 0x7f8d7846f740
status =
#11 0x55e62bffc01c in ap_content_length_filter (f=0x7f8d78472ca8,
b=0x7f8d7864ba90) at protocol.c:1713
r = 0x7f8d78471750
ctx = 0x7f8d7864bc90
e = 0x7f8d7864ba98
eblock = (unknown: 2159446012)
#12 0x55e62c04853d in deflate_out_filter (f=0x7f8d7864b650,
bb=0x7f8d7864b8f8) at mod_deflate.c:961
rv = -512
r = 0x7f8d78471750
ctx = 0x7f8d7864b9a8
len = 8096
blen = 1157663
data = 0x7f8d7e40b000 "/*! jQuery v2.1.4 | (c) 2005, 2015 jQuery
Foundation, Inc. | jquery.org/license
*/\n!function(a,b){\"object\"==typeof module&&\"object\"==typeof
module.exports?module.exports=a.document?b(a,!0):function(a)"...
c = 0x55e62c7a7498
#13 0x55e62c044b1d in filter_harness (f=0x7f8d7846bc28, bb=0x80) at
mod_filter.c:323
ret = -512
cachecontrol = 0xfe00 >>> at address 0xfe00>
filter = 0xe
#14 0x55e62c002242 in default_handler (r=0x7f8d78471750) at core.c:4746
c = 0x7f8d7846b720
bb = 0x7f8d7864b8f8
e = 0xfe00
d = 0x7f8d7864dfa0
fd = 0x7f8d7864b7b8
bld_content_md5 = 2019866872
#15 0x55e62c014130 in ap_run_handler (r=r@entry=0x7f8d78471750) at
config.c:170
pHook =
n = 11
rv = -1
#16 0x55e62c014679 in ap_invoke_handler (r=0x7f8d78471750) at
config.c:434
handler =
p =
result =
old_handler = 0x0
ignore =
#17 0x55e62c04dcb2 in ap_process_async_request (r=0x7f8d78471750) at
http_request.c:436
c = 0x7f8d7846b720
acce
Re: hanging apache httpd processes
In transit. Just some minutes away...
> Am 18.04.2017 um 15:06 schrieb Stefan Priebe - Profihost AG
> :
>
>
> Am 18.04.2017 um 15:03 schrieb Stefan Eissing:
>> Stefan,
>>
>> that is a 1.10.0, right? That was the first version without nested locking
>> and I fixed 2 possible dead locks in 1.10.1.
>>
>> I am about to release a 1.10.2 with added conformity checks and a fix for
>> client omitting EOF flags. Could you give that one a try?
>
> Yes sure where is 1.10.2?
>
> Stefan
>
>>
>> -Stefan
>>
>>> Am 18.04.2017 um 14:57 schrieb Stefan Priebe - Profihost AG
>>> :
>>>
>>> Hi,
>>>
>>> i saw that all of them are still serving one h2 connection.
>>>
>>> server-status:
>>> 0-3 32375 42/64/181 G 30.09 1020776 214 1285.1 2.40
>>> 5.81
>>> h081217236127.dyn.cm.kabsi.at h2 :443GET
>>> /wp-content/uploads/Bloglr21-8003asd.jpg HTTP/2.0
>>>
>>> And they all have Mode of operation => G
>>>
>>> thread apply all bt full shows:
>>> #0 0x7f8e80b687fc in __lll_lock_wait () from
>>> /lib/x86_64-linux-gnu/libpthread.so.0
>>> No symbol table info available.
>>> #1 0x7f8e80b6b6f8 in _L_cond_lock_886 () from
>>> /lib/x86_64-linux-gnu/libpthread.so.0
>>> No symbol table info available.
>>> #2 0x7f8e80b6b464 in __pthread_mutex_cond_lock () from
>>> /lib/x86_64-linux-gnu/libpthread.so.0
>>> No symbol table info available.
>>> #3 0x7f8e80b6650a in pthread_cond_timedwait@@GLIBC_2.3.2 () from
>>> /lib/x86_64-linux-gnu/libpthread.so.0
>>> No symbol table info available.
>>> #4 0x7f8e80d92322 in apr_thread_cond_timedwait
>>> (cond=0x7f8d7846bc70, mutex=0x7f8d7846bc20, timeout=6000) at
>>> locks/unix/thread_cond.c:89
>>> rv =
>>> then =
>>> abstime = {tv_sec = 1491843095, tv_nsec = 700198000}
>>> #5 0x55e62c0951c5 in r_wait_space (premain=,
>>> pbl=0x7f8de853da60, block=APR_BLOCK_READ, beam=0x7f8d7846bb08) at
>>> h2_bucket_beam.c:337
>>> status =
>>> #6 append_bucket (pbl=0x7f8de853da60, block=APR_BLOCK_READ,
>>> b=0x7f8d7846dd38, beam=0x7f8d7846bb08) at h2_bucket_beam.c:776
>>> data = 0x7f8de853dac8 "\240\037"
>>> len = 0
>>> space_left = 0
>>> status =
>>> #7 h2_beam_send (beam=0x7f8d7846bb08, sender_bb=0x7f8d7864ba90,
>>> block=APR_BLOCK_READ) at h2_bucket_beam.c:906
>>> force_report = 1
>>> b = 0x7f8d7846dd38
>>> status = 0
>>> bl = {mutex = 0x7f8d7846bc20, leave = 0x55e62c094250
>>> , leave_ctx = 0x0}
>>> #8 0x55e62c08f3aa in send_out (task=0x7f8d7846f740,
>>> bb=0x7f8d7864ba90, block=1) at h2_task.c:100
>>> written = 8096
>>> left = 140245585033024
>>> status = 1
>>> #9 0x55e62c08f976 in slave_out (task=0x7f8d7846f740,
>>> f=0x7f8d7846bdd8, bb=0x7f8d7864ba90) at h2_task.c:177
>>> status = -512
>>> flush = 0
>>> blocking = 1
>>> #10 0x55e62c08fbfd in h2_filter_slave_output (filter=0x7f8d7846bdd8,
>>> brigade=0x7f8d7864ba90) at h2_task.c:370
>>> task = 0x7f8d7846f740
>>> status =
>>> #11 0x55e62bffc01c in ap_content_length_filter (f=0x7f8d78472ca8,
>>> b=0x7f8d7864ba90) at protocol.c:1713
>>> r = 0x7f8d78471750
>>> ctx = 0x7f8d7864bc90
>>> e = 0x7f8d7864ba98
>>> eblock = (unknown: 2159446012)
>>> #12 0x55e62c04853d in deflate_out_filter (f=0x7f8d7864b650,
>>> bb=0x7f8d7864b8f8) at mod_deflate.c:961
>>> rv = -512
>>> r = 0x7f8d78471750
>>> ctx = 0x7f8d7864b9a8
>>> len = 8096
>>> blen = 1157663
>>> data = 0x7f8d7e40b000 "/*! jQuery v2.1.4 | (c) 2005, 2015 jQuery
>>> Foundation, Inc. | jquery.org/license
>>> */\n!function(a,b){\"object\"==typeof module&&\"object\"==typeof
>>> module.exports?module.exports=a.document?b(a,!0):function(a)"...
>>> c = 0x55e62c7a7498
>>> #13 0x55e62c044b1d in filter_harness (f=0x7f8d7846bc28, bb=0x80) at
>>> mod_filter.c:323
>>> ret = -512
>>> cachecontrol = 0xfe00 >> at address 0xfe00>
>>> filter = 0xe
>>> #14 0x55e62c002242 in default_handler (r=0x7f8d78471750) at core.c:4746
>>> c = 0x7f8d7846b720
>>> bb = 0x7f8d7864b8f8
>>> e = 0xfe00
>>> d = 0x7f8d7864dfa0
>>> fd = 0x7f8d7864b7b8
>>> bld_content_md5 = 2019866872
>>> #15 0x55e62c014130 in ap_run_handler (r=r@entry=0x7f8d78471750) at
>>> config.c:170
>>> pHook =
>>> n = 11
>>> rv = -1
>>> #16 0x55e62c014679 in ap_invoke_handler (r=0x7f8d78471750) at
>>> config.c:434
>>> handler =
>>> p =
>>> result =
>>> old_handler = 0x0
>>> ignore =
>>> #17 0x55e62c04dcb2 in ap_process_async_request (r=0x7f8d78471750) at
>>> http_request.c:436
>>> c = 0x7f8d7846b720
>>> access_status = 0
>>> #18 0x55e62c04de50 in ap_process_request (r=0x7f8d78471750) at
>>> http_request.c:471
>>> bb = 0x7f8d7846bd80
>>> c = 0x7f8d7846b720
>>> rv = -512
>>> #19 0x55e62c08ec
Re: hanging apache httpd processes
Am 18.04.2017 um 15:03 schrieb Stefan Eissing:
> Stefan,
>
> that is a 1.10.0, right? That was the first version without nested locking
> and I fixed 2 possible dead locks in 1.10.1.
>
> I am about to release a 1.10.2 with added conformity checks and a fix for
> client omitting EOF flags. Could you give that one a try?
Yes sure where is 1.10.2?
Stefan
>
> -Stefan
>
>> Am 18.04.2017 um 14:57 schrieb Stefan Priebe - Profihost AG
>> :
>>
>> Hi,
>>
>> i saw that all of them are still serving one h2 connection.
>>
>> server-status:
>> 0-3 32375 42/64/181 G 30.09 1020776 214 1285.1 2.40
>> 5.81
>> h081217236127.dyn.cm.kabsi.ath2 :443GET
>> /wp-content/uploads/Bloglr21-8003asd.jpg HTTP/2.0
>>
>> And they all have Mode of operation => G
>>
>> thread apply all bt full shows:
>> #0 0x7f8e80b687fc in __lll_lock_wait () from
>> /lib/x86_64-linux-gnu/libpthread.so.0
>> No symbol table info available.
>> #1 0x7f8e80b6b6f8 in _L_cond_lock_886 () from
>> /lib/x86_64-linux-gnu/libpthread.so.0
>> No symbol table info available.
>> #2 0x7f8e80b6b464 in __pthread_mutex_cond_lock () from
>> /lib/x86_64-linux-gnu/libpthread.so.0
>> No symbol table info available.
>> #3 0x7f8e80b6650a in pthread_cond_timedwait@@GLIBC_2.3.2 () from
>> /lib/x86_64-linux-gnu/libpthread.so.0
>> No symbol table info available.
>> #4 0x7f8e80d92322 in apr_thread_cond_timedwait
>> (cond=0x7f8d7846bc70, mutex=0x7f8d7846bc20, timeout=6000) at
>> locks/unix/thread_cond.c:89
>>rv =
>>then =
>>abstime = {tv_sec = 1491843095, tv_nsec = 700198000}
>> #5 0x55e62c0951c5 in r_wait_space (premain=,
>> pbl=0x7f8de853da60, block=APR_BLOCK_READ, beam=0x7f8d7846bb08) at
>> h2_bucket_beam.c:337
>>status =
>> #6 append_bucket (pbl=0x7f8de853da60, block=APR_BLOCK_READ,
>> b=0x7f8d7846dd38, beam=0x7f8d7846bb08) at h2_bucket_beam.c:776
>>data = 0x7f8de853dac8 "\240\037"
>>len = 0
>>space_left = 0
>>status =
>> #7 h2_beam_send (beam=0x7f8d7846bb08, sender_bb=0x7f8d7864ba90,
>> block=APR_BLOCK_READ) at h2_bucket_beam.c:906
>>force_report = 1
>>b = 0x7f8d7846dd38
>>status = 0
>>bl = {mutex = 0x7f8d7846bc20, leave = 0x55e62c094250
>> , leave_ctx = 0x0}
>> #8 0x55e62c08f3aa in send_out (task=0x7f8d7846f740,
>> bb=0x7f8d7864ba90, block=1) at h2_task.c:100
>>written = 8096
>>left = 140245585033024
>>status = 1
>> #9 0x55e62c08f976 in slave_out (task=0x7f8d7846f740,
>> f=0x7f8d7846bdd8, bb=0x7f8d7864ba90) at h2_task.c:177
>>status = -512
>>flush = 0
>>blocking = 1
>> #10 0x55e62c08fbfd in h2_filter_slave_output (filter=0x7f8d7846bdd8,
>> brigade=0x7f8d7864ba90) at h2_task.c:370
>>task = 0x7f8d7846f740
>>status =
>> #11 0x55e62bffc01c in ap_content_length_filter (f=0x7f8d78472ca8,
>> b=0x7f8d7864ba90) at protocol.c:1713
>>r = 0x7f8d78471750
>>ctx = 0x7f8d7864bc90
>>e = 0x7f8d7864ba98
>>eblock = (unknown: 2159446012)
>> #12 0x55e62c04853d in deflate_out_filter (f=0x7f8d7864b650,
>> bb=0x7f8d7864b8f8) at mod_deflate.c:961
>>rv = -512
>>r = 0x7f8d78471750
>>ctx = 0x7f8d7864b9a8
>>len = 8096
>>blen = 1157663
>>data = 0x7f8d7e40b000 "/*! jQuery v2.1.4 | (c) 2005, 2015 jQuery
>> Foundation, Inc. | jquery.org/license
>> */\n!function(a,b){\"object\"==typeof module&&\"object\"==typeof
>> module.exports?module.exports=a.document?b(a,!0):function(a)"...
>>c = 0x55e62c7a7498
>> #13 0x55e62c044b1d in filter_harness (f=0x7f8d7846bc28, bb=0x80) at
>> mod_filter.c:323
>>ret = -512
>>cachecontrol = 0xfe00 > at address 0xfe00>
>>filter = 0xe
>> #14 0x55e62c002242 in default_handler (r=0x7f8d78471750) at core.c:4746
>>c = 0x7f8d7846b720
>>bb = 0x7f8d7864b8f8
>>e = 0xfe00
>>d = 0x7f8d7864dfa0
>>fd = 0x7f8d7864b7b8
>>bld_content_md5 = 2019866872
>> #15 0x55e62c014130 in ap_run_handler (r=r@entry=0x7f8d78471750) at
>> config.c:170
>>pHook =
>>n = 11
>>rv = -1
>> #16 0x55e62c014679 in ap_invoke_handler (r=0x7f8d78471750) at
>> config.c:434
>>handler =
>>p =
>>result =
>>old_handler = 0x0
>>ignore =
>> #17 0x55e62c04dcb2 in ap_process_async_request (r=0x7f8d78471750) at
>> http_request.c:436
>>c = 0x7f8d7846b720
>>access_status = 0
>> #18 0x55e62c04de50 in ap_process_request (r=0x7f8d78471750) at
>> http_request.c:471
>>bb = 0x7f8d7846bd80
>>c = 0x7f8d7846b720
>>rv = -512
>> #19 0x55e62c08ec32 in h2_task_process_request (c=,
>> task=) at h2_task.c:666
>>req = 0x7f8d78471750
>>cs = 0x7f8d7846bd80
>>r = 0x7f8d78471750
>> #20 h2_task_process_conn (c=0x7f8d7846bc28) at
Re: hanging apache httpd processes
Stefan,
that is a 1.10.0, right? That was the first version without nested locking and
I fixed 2 possible dead locks in 1.10.1.
I am about to release a 1.10.2 with added conformity checks and a fix for
client omitting EOF flags. Could you give that one a try?
-Stefan
> Am 18.04.2017 um 14:57 schrieb Stefan Priebe - Profihost AG
> :
>
> Hi,
>
> i saw that all of them are still serving one h2 connection.
>
> server-status:
> 0-3 32375 42/64/181 G 30.09 1020776 214 1285.1 2.40
> 5.81
> h081217236127.dyn.cm.kabsi.at h2 :443GET
> /wp-content/uploads/Bloglr21-8003asd.jpg HTTP/2.0
>
> And they all have Mode of operation => G
>
> thread apply all bt full shows:
> #0 0x7f8e80b687fc in __lll_lock_wait () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> No symbol table info available.
> #1 0x7f8e80b6b6f8 in _L_cond_lock_886 () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> No symbol table info available.
> #2 0x7f8e80b6b464 in __pthread_mutex_cond_lock () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> No symbol table info available.
> #3 0x7f8e80b6650a in pthread_cond_timedwait@@GLIBC_2.3.2 () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> No symbol table info available.
> #4 0x7f8e80d92322 in apr_thread_cond_timedwait
> (cond=0x7f8d7846bc70, mutex=0x7f8d7846bc20, timeout=6000) at
> locks/unix/thread_cond.c:89
>rv =
>then =
>abstime = {tv_sec = 1491843095, tv_nsec = 700198000}
> #5 0x55e62c0951c5 in r_wait_space (premain=,
> pbl=0x7f8de853da60, block=APR_BLOCK_READ, beam=0x7f8d7846bb08) at
> h2_bucket_beam.c:337
>status =
> #6 append_bucket (pbl=0x7f8de853da60, block=APR_BLOCK_READ,
> b=0x7f8d7846dd38, beam=0x7f8d7846bb08) at h2_bucket_beam.c:776
>data = 0x7f8de853dac8 "\240\037"
>len = 0
>space_left = 0
>status =
> #7 h2_beam_send (beam=0x7f8d7846bb08, sender_bb=0x7f8d7864ba90,
> block=APR_BLOCK_READ) at h2_bucket_beam.c:906
>force_report = 1
>b = 0x7f8d7846dd38
>status = 0
>bl = {mutex = 0x7f8d7846bc20, leave = 0x55e62c094250
> , leave_ctx = 0x0}
> #8 0x55e62c08f3aa in send_out (task=0x7f8d7846f740,
> bb=0x7f8d7864ba90, block=1) at h2_task.c:100
>written = 8096
>left = 140245585033024
>status = 1
> #9 0x55e62c08f976 in slave_out (task=0x7f8d7846f740,
> f=0x7f8d7846bdd8, bb=0x7f8d7864ba90) at h2_task.c:177
>status = -512
>flush = 0
>blocking = 1
> #10 0x55e62c08fbfd in h2_filter_slave_output (filter=0x7f8d7846bdd8,
> brigade=0x7f8d7864ba90) at h2_task.c:370
>task = 0x7f8d7846f740
>status =
> #11 0x55e62bffc01c in ap_content_length_filter (f=0x7f8d78472ca8,
> b=0x7f8d7864ba90) at protocol.c:1713
>r = 0x7f8d78471750
>ctx = 0x7f8d7864bc90
>e = 0x7f8d7864ba98
>eblock = (unknown: 2159446012)
> #12 0x55e62c04853d in deflate_out_filter (f=0x7f8d7864b650,
> bb=0x7f8d7864b8f8) at mod_deflate.c:961
>rv = -512
>r = 0x7f8d78471750
>ctx = 0x7f8d7864b9a8
>len = 8096
>blen = 1157663
>data = 0x7f8d7e40b000 "/*! jQuery v2.1.4 | (c) 2005, 2015 jQuery
> Foundation, Inc. | jquery.org/license
> */\n!function(a,b){\"object\"==typeof module&&\"object\"==typeof
> module.exports?module.exports=a.document?b(a,!0):function(a)"...
>c = 0x55e62c7a7498
> #13 0x55e62c044b1d in filter_harness (f=0x7f8d7846bc28, bb=0x80) at
> mod_filter.c:323
>ret = -512
>cachecontrol = 0xfe00 at address 0xfe00>
>filter = 0xe
> #14 0x55e62c002242 in default_handler (r=0x7f8d78471750) at core.c:4746
>c = 0x7f8d7846b720
>bb = 0x7f8d7864b8f8
>e = 0xfe00
>d = 0x7f8d7864dfa0
>fd = 0x7f8d7864b7b8
>bld_content_md5 = 2019866872
> #15 0x55e62c014130 in ap_run_handler (r=r@entry=0x7f8d78471750) at
> config.c:170
>pHook =
>n = 11
>rv = -1
> #16 0x55e62c014679 in ap_invoke_handler (r=0x7f8d78471750) at
> config.c:434
>handler =
>p =
>result =
>old_handler = 0x0
>ignore =
> #17 0x55e62c04dcb2 in ap_process_async_request (r=0x7f8d78471750) at
> http_request.c:436
>c = 0x7f8d7846b720
>access_status = 0
> #18 0x55e62c04de50 in ap_process_request (r=0x7f8d78471750) at
> http_request.c:471
>bb = 0x7f8d7846bd80
>c = 0x7f8d7846b720
>rv = -512
> #19 0x55e62c08ec32 in h2_task_process_request (c=,
> task=) at h2_task.c:666
>req = 0x7f8d78471750
>cs = 0x7f8d7846bd80
>r = 0x7f8d78471750
> #20 h2_task_process_conn (c=0x7f8d7846bc28) at h2_task.c:713
>ctx = 0x7f8d7846bd80
> #21 0x55e62c01e130 in ap_run_process_connection (c=0x7f8d7846b720)
> at connection.c:42
>pHook =
>n = 0
>rv = -1
> #22 0x55e62c09012b in h2_task_do
Re: hanging apache httpd processes
Hi,
i saw that all of them are still serving one h2 connection.
server-status:
0-3 32375 42/64/181 G 30.09 1020776 214 1285.1 2.40
5.81
h081217236127.dyn.cm.kabsi.at h2 :443GET
/wp-content/uploads/Bloglr21-8003asd.jpg HTTP/2.0
And they all have Mode of operation => G
thread apply all bt full shows:
#0 0x7f8e80b687fc in __lll_lock_wait () from
/lib/x86_64-linux-gnu/libpthread.so.0
No symbol table info available.
#1 0x7f8e80b6b6f8 in _L_cond_lock_886 () from
/lib/x86_64-linux-gnu/libpthread.so.0
No symbol table info available.
#2 0x7f8e80b6b464 in __pthread_mutex_cond_lock () from
/lib/x86_64-linux-gnu/libpthread.so.0
No symbol table info available.
#3 0x7f8e80b6650a in pthread_cond_timedwait@@GLIBC_2.3.2 () from
/lib/x86_64-linux-gnu/libpthread.so.0
No symbol table info available.
#4 0x7f8e80d92322 in apr_thread_cond_timedwait
(cond=0x7f8d7846bc70, mutex=0x7f8d7846bc20, timeout=6000) at
locks/unix/thread_cond.c:89
rv =
then =
abstime = {tv_sec = 1491843095, tv_nsec = 700198000}
#5 0x55e62c0951c5 in r_wait_space (premain=,
pbl=0x7f8de853da60, block=APR_BLOCK_READ, beam=0x7f8d7846bb08) at
h2_bucket_beam.c:337
status =
#6 append_bucket (pbl=0x7f8de853da60, block=APR_BLOCK_READ,
b=0x7f8d7846dd38, beam=0x7f8d7846bb08) at h2_bucket_beam.c:776
data = 0x7f8de853dac8 "\240\037"
len = 0
space_left = 0
status =
#7 h2_beam_send (beam=0x7f8d7846bb08, sender_bb=0x7f8d7864ba90,
block=APR_BLOCK_READ) at h2_bucket_beam.c:906
force_report = 1
b = 0x7f8d7846dd38
status = 0
bl = {mutex = 0x7f8d7846bc20, leave = 0x55e62c094250
, leave_ctx = 0x0}
#8 0x55e62c08f3aa in send_out (task=0x7f8d7846f740,
bb=0x7f8d7864ba90, block=1) at h2_task.c:100
written = 8096
left = 140245585033024
status = 1
#9 0x55e62c08f976 in slave_out (task=0x7f8d7846f740,
f=0x7f8d7846bdd8, bb=0x7f8d7864ba90) at h2_task.c:177
status = -512
flush = 0
blocking = 1
#10 0x55e62c08fbfd in h2_filter_slave_output (filter=0x7f8d7846bdd8,
brigade=0x7f8d7864ba90) at h2_task.c:370
task = 0x7f8d7846f740
status =
#11 0x55e62bffc01c in ap_content_length_filter (f=0x7f8d78472ca8,
b=0x7f8d7864ba90) at protocol.c:1713
r = 0x7f8d78471750
ctx = 0x7f8d7864bc90
e = 0x7f8d7864ba98
eblock = (unknown: 2159446012)
#12 0x55e62c04853d in deflate_out_filter (f=0x7f8d7864b650,
bb=0x7f8d7864b8f8) at mod_deflate.c:961
rv = -512
r = 0x7f8d78471750
ctx = 0x7f8d7864b9a8
len = 8096
blen = 1157663
data = 0x7f8d7e40b000 "/*! jQuery v2.1.4 | (c) 2005, 2015 jQuery
Foundation, Inc. | jquery.org/license
*/\n!function(a,b){\"object\"==typeof module&&\"object\"==typeof
module.exports?module.exports=a.document?b(a,!0):function(a)"...
c = 0x55e62c7a7498
#13 0x55e62c044b1d in filter_harness (f=0x7f8d7846bc28, bb=0x80) at
mod_filter.c:323
ret = -512
cachecontrol = 0xfe00
filter = 0xe
#14 0x55e62c002242 in default_handler (r=0x7f8d78471750) at core.c:4746
c = 0x7f8d7846b720
bb = 0x7f8d7864b8f8
e = 0xfe00
d = 0x7f8d7864dfa0
fd = 0x7f8d7864b7b8
bld_content_md5 = 2019866872
#15 0x55e62c014130 in ap_run_handler (r=r@entry=0x7f8d78471750) at
config.c:170
pHook =
n = 11
rv = -1
#16 0x55e62c014679 in ap_invoke_handler (r=0x7f8d78471750) at
config.c:434
handler =
p =
result =
old_handler = 0x0
ignore =
#17 0x55e62c04dcb2 in ap_process_async_request (r=0x7f8d78471750) at
http_request.c:436
c = 0x7f8d7846b720
access_status = 0
#18 0x55e62c04de50 in ap_process_request (r=0x7f8d78471750) at
http_request.c:471
bb = 0x7f8d7846bd80
c = 0x7f8d7846b720
rv = -512
#19 0x55e62c08ec32 in h2_task_process_request (c=,
task=) at h2_task.c:666
req = 0x7f8d78471750
cs = 0x7f8d7846bd80
r = 0x7f8d78471750
#20 h2_task_process_conn (c=0x7f8d7846bc28) at h2_task.c:713
ctx = 0x7f8d7846bd80
#21 0x55e62c01e130 in ap_run_process_connection (c=0x7f8d7846b720)
at connection.c:42
pHook =
n = 0
rv = -1
#22 0x55e62c09012b in h2_task_do (task=0x7f8d7846f740,
thread=0x55e62cb0ce70, worker_id=0) at h2_task.c:623
c = 0x7f8d7846b720
#23 0x55e62c093619 in slot_run (thread=0x55e62cb0ce70,
wctx=0x55e62c8b9268) at h2_workers.c:217
slot = 0x55e62c8b9268
#24 0x7f8e80b620a4 in start_thread () from
/lib/x86_64-linux-gnu/libpthread.so.0
No symbol table info available.
#25 0x7f8e8089762d in clone () from /lib/x86_64-linux-gnu/libc.so.6
No symbol table info available.
Thread 3 (Thread 0x7f8da678c700 (LWP 28251)):
#0 0x7f8e80b687fc in __lll_lock_wait () from
/lib/x86_64-li
Re: hanging apache httpd processes
What Eric said. With the changes in http2 worker scheduling, if I introduced a bug in child exit there, I'd expect it to trigger via workers_pool_cleanup() in h2_workers.c -Stefan > Am 18.04.2017 um 14:47 schrieb Eric Covener : > > On Tue, Apr 18, 2017 at 8:43 AM, Stefan Priebe - Profihost AG > wrote: >> bt of such a process shows: >> (gdb) bt >> #0 0x7f5df74f64db in pthread_join () from >> /lib/x86_64-linux-gnu/libpthread.so.0 > > Need to see the other threads in the process, this one is just waiting > for the others to complete. > > "thread apply all bt full" > > > -- > Eric Covener > [email protected]
Re: hanging apache httpd processes
On Tue, Apr 18, 2017 at 8:43 AM, Stefan Priebe - Profihost AG wrote: > bt of such a process shows: > (gdb) bt > #0 0x7f5df74f64db in pthread_join () from > /lib/x86_64-linux-gnu/libpthread.so.0 Need to see the other threads in the process, this one is just waiting for the others to complete. "thread apply all bt full" -- Eric Covener [email protected]
hanging apache httpd processes
Hi, not sure whether this is related to mod_http2 v1.10.0 or is something else. I've seen two servers where old httpd processes get stuck. server-status looks like this: SlotPID StoppingConnections Threads Async connections total accepting busyidlewriting keep-alive closing 0 27399 yes (old gen) 3 no 0 0 0 0 0 1 879 yes (old gen) 1 no 0 0 0 0 0 2 11609 yes (old gen) 2 no 0 0 0 0 0 3 9142yes (old gen) 2 no 0 0 0 0 0 4 11098 yes (old gen) 3 no 0 0 0 0 0 5 27400 yes (old gen) 1 no 0 0 0 0 0 6 16445 yes (old gen) 1 no 0 0 0 0 0 7 16446 yes (old gen) 1 no 0 0 0 0 0 8 7388yes (old gen) 4 no 0 0 0 0 0 9 24072 yes (old gen) 2 no 0 0 0 0 0 10 28673 yes (old gen) 2 no 0 0 0 0 0 11 7389yes (old gen) 3 no 0 0 0 0 0 12 11075 yes (old gen) 8 no 0 0 0 0 0 13 11076 yes (old gen) 6 no 0 0 0 0 0 14 26613 yes (old gen) 5 no 0 0 0 0 0 15 737 no 1 yes 8 192 0 0 1 16 739 no 3 yes 13 187 0 0 1 17 24059 yes (old gen) 1 no 0 0 0 0 0 18 26002 yes (old gen) 7 no 0 0 0 0 0 19 26003 yes (old gen) 4 no 0 0 0 0 0 20 2473yes (old gen) 2 no 0 0 0 0 0 21 3010yes (old gen) 4 no 0 0 0 0 0 22 8847yes (old gen) 3 no 0 0 0 0 0 Sum 23 21 69 21 379 0 0 2 bt of such a process shows: (gdb) bt #0 0x7f5df74f64db in pthread_join () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x7f5df77314ab in apr_thread_join (retval=0x7ffc0878f424, thd=0x7f5d580dc7c8) at threadproc/unix/thread.c:217 #2 0x555c1dc9deb3 in join_workers (listener=0x7f5d580dc7c8, threads=0x555c1f3d4310) at event.c:2376 #3 0x555c1dc9e25a in child_main (child_num_arg=0, child_bucket=524108560) at event.c:2574 #4 0x555c1dd68fe0 in make_child (s=0x7f5cf673c9d0, slot=0, bucket=0) at event.c:2646 #5 0x555c1dd6950c in server_main_loop (num_buckets=, remaining_children_to_start=) at event.c:2916 #6 event_run (_pconf=0x7f5cf673c9d0, plog=0x0, s=0xc8) at event.c:3035 #7 0x555c1dca642e in ap_run_mpm (pconf=0x555c1f031138, plog=0x555c1f072bb8, s=0x555c1f0625a8) at mpm_common.c:94 #8 0x555c1dc9ed15 in main (argc=2, argv=0x7ffc0878f828) at main.c:783 An strace is like this one: strace -ff -p 32375 Process 32375 attached with 303 threads [pid 856] futex(0x7f5ce8003c60, FUTEX_WAIT_PRIVATE, 2, NULL [pid 486] futex(0x555c1f3b17d4, FUTEX_WAIT_PRIVATE, 1, NULL [pid 857] epoll_wait(14, [pid 484] futex(0x555c1f3b174c, FUTEX_WAIT_PRIVATE, 1, NULL [pid 482] futex(0x555c1f3b16c4, FUTEX_WAIT_PRIVATE, 1, NULL [pid 481] futex(0x555c1f3b163c, FUTEX_WAIT_PRIVATE, 1, NULL ... Greets, Stefan
