Re: hanging apache httpd processes due to mod_h2
looks good so war. Will wait some more days. Greets, Stefan Am 19.04.2017 um 11:09 schrieb Stefan Eissing: > Thanks, the backtraces really helped. Could you try the following patch? > > -Stefan > > > > >> Am 19.04.2017 um 08:32 schrieb Stefan Priebe - Profihost AG >>: >> >> Hi Stefan, >> >> this is now with v1.10.2 but it does not help. >> >> PID old gen: >> >> 215608 yes (old gen) 2 no 0 0 0 0 >> 0 >> >> >> requests in G state: >> >> 2-2 15608 4/36/36 G 252.75 30695 41 22.10.870.87 >> A.B.C.D h2 XYZ:443 >> GET /themes/Frontend/Responsive/frontend/_public/src/js/vendors >> >> 2-2 15608 2/5/5 G 69.62 39645 36 104.0 0.180.18 >> A.B.C.D h2 XYZ:443 GET >> /web/cache/1492526985_9e591f3c4f420ebca36f3c1be30bd5ee.css >> >> thread all bt: >> https://pastebin.com/raw/Cq9ymt9u >> >> I think the interesting thread is: >> Thread 6 >> >> Greets, >> Stefan >> >> Am 18.04.2017 um 15:06 schrieb Stefan Priebe - Profihost AG: >>> >>> Am 18.04.2017 um 15:03 schrieb Stefan Eissing: Stefan, that is a 1.10.0, right? That was the first version without nested locking and I fixed 2 possible dead locks in 1.10.1. I am about to release a 1.10.2 with added conformity checks and a fix for client omitting EOF flags. Could you give that one a try? >>> >>> Yes sure where is 1.10.2? >>> >>> Stefan >>> -Stefan > Am 18.04.2017 um 14:57 schrieb Stefan Priebe - Profihost AG > : > > Hi, > > i saw that all of them are still serving one h2 connection. > > server-status: > 0-3 32375 42/64/181 G 30.09 1020776 214 1285.1 > 2.405.81 > h081217236127.dyn.cm.kabsi.at h2 :443GET > /wp-content/uploads/Bloglr21-8003asd.jpg HTTP/2.0 > > And they all have Mode of operation => G > > thread apply all bt full shows: > #0 0x7f8e80b687fc in __lll_lock_wait () from > /lib/x86_64-linux-gnu/libpthread.so.0 > No symbol table info available. > #1 0x7f8e80b6b6f8 in _L_cond_lock_886 () from > /lib/x86_64-linux-gnu/libpthread.so.0 > No symbol table info available. > #2 0x7f8e80b6b464 in __pthread_mutex_cond_lock () from > /lib/x86_64-linux-gnu/libpthread.so.0 > No symbol table info available. > #3 0x7f8e80b6650a in pthread_cond_timedwait@@GLIBC_2.3.2 () from > /lib/x86_64-linux-gnu/libpthread.so.0 > No symbol table info available. > #4 0x7f8e80d92322 in apr_thread_cond_timedwait > (cond=0x7f8d7846bc70, mutex=0x7f8d7846bc20, timeout=6000) at > locks/unix/thread_cond.c:89 > rv = > then = > abstime = {tv_sec = 1491843095, tv_nsec = 700198000} > #5 0x55e62c0951c5 in r_wait_space (premain=, > pbl=0x7f8de853da60, block=APR_BLOCK_READ, beam=0x7f8d7846bb08) at > h2_bucket_beam.c:337 > status = > #6 append_bucket (pbl=0x7f8de853da60, block=APR_BLOCK_READ, > b=0x7f8d7846dd38, beam=0x7f8d7846bb08) at h2_bucket_beam.c:776 > data = 0x7f8de853dac8 "\240\037" > len = 0 > space_left = 0 > status = > #7 h2_beam_send (beam=0x7f8d7846bb08, sender_bb=0x7f8d7864ba90, > block=APR_BLOCK_READ) at h2_bucket_beam.c:906 > force_report = 1 > b = 0x7f8d7846dd38 > status = 0 > bl = {mutex = 0x7f8d7846bc20, leave = 0x55e62c094250 > , leave_ctx = 0x0} > #8 0x55e62c08f3aa in send_out (task=0x7f8d7846f740, > bb=0x7f8d7864ba90, block=1) at h2_task.c:100 > written = 8096 > left = 140245585033024 > status = 1 > #9 0x55e62c08f976 in slave_out (task=0x7f8d7846f740, > f=0x7f8d7846bdd8, bb=0x7f8d7864ba90) at h2_task.c:177 > status = -512 > flush = 0 > blocking = 1 > #10 0x55e62c08fbfd in h2_filter_slave_output (filter=0x7f8d7846bdd8, > brigade=0x7f8d7864ba90) at h2_task.c:370 > task = 0x7f8d7846f740 > status = > #11 0x55e62bffc01c in ap_content_length_filter (f=0x7f8d78472ca8, > b=0x7f8d7864ba90) at protocol.c:1713 > r = 0x7f8d78471750 > ctx = 0x7f8d7864bc90 > e = 0x7f8d7864ba98 > eblock = (unknown: 2159446012) > #12 0x55e62c04853d in deflate_out_filter (f=0x7f8d7864b650, > bb=0x7f8d7864b8f8) at mod_deflate.c:961 > rv = -512 > r = 0x7f8d78471750 > ctx = 0x7f8d7864b9a8 > len = 8096 > blen = 1157663 > data = 0x7f8d7e40b000 "/*! jQuery v2.1.4 | (c) 2005, 2015 jQuery > Foundation, Inc. | jquery.org/license > */\n!function(a,b){\"object\"==typeof module&&\"object\"==typeof > module.exports?module.exports=a.document?b(a,!0):function(a)"... > c = 0x55e62c7a7498 > #13
Re: hanging apache httpd processes due to mod_h2
Thanks, the backtraces really helped. Could you try the following patch? -Stefan h2_beam_locks_v1.diff Description: Binary data > Am 19.04.2017 um 08:32 schrieb Stefan Priebe - Profihost AG >: > > Hi Stefan, > > this is now with v1.10.2 but it does not help. > > PID old gen: > > 2 15608 yes (old gen) 2 no 0 0 0 0 > 0 > > > requests in G state: > > 2-2 15608 4/36/36 G 252.75 30695 41 22.10.870.87 > A.B.C.D h2 XYZ:443 > GET /themes/Frontend/Responsive/frontend/_public/src/js/vendors > > 2-2 15608 2/5/5 G 69.62 39645 36 104.0 0.180.18 > A.B.C.D h2 XYZ:443 GET > /web/cache/1492526985_9e591f3c4f420ebca36f3c1be30bd5ee.css > > thread all bt: > https://pastebin.com/raw/Cq9ymt9u > > I think the interesting thread is: > Thread 6 > > Greets, > Stefan > > Am 18.04.2017 um 15:06 schrieb Stefan Priebe - Profihost AG: >> >> Am 18.04.2017 um 15:03 schrieb Stefan Eissing: >>> Stefan, >>> >>> that is a 1.10.0, right? That was the first version without nested locking >>> and I fixed 2 possible dead locks in 1.10.1. >>> >>> I am about to release a 1.10.2 with added conformity checks and a fix for >>> client omitting EOF flags. Could you give that one a try? >> >> Yes sure where is 1.10.2? >> >> Stefan >> >>> >>> -Stefan >>> Am 18.04.2017 um 14:57 schrieb Stefan Priebe - Profihost AG : Hi, i saw that all of them are still serving one h2 connection. server-status: 0-332375 42/64/181 G 30.09 1020776 214 1285.1 2.405.81 h081217236127.dyn.cm.kabsi.at h2 :443GET /wp-content/uploads/Bloglr21-8003asd.jpg HTTP/2.0 And they all have Mode of operation => G thread apply all bt full shows: #0 0x7f8e80b687fc in __lll_lock_wait () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #1 0x7f8e80b6b6f8 in _L_cond_lock_886 () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #2 0x7f8e80b6b464 in __pthread_mutex_cond_lock () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #3 0x7f8e80b6650a in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #4 0x7f8e80d92322 in apr_thread_cond_timedwait (cond=0x7f8d7846bc70, mutex=0x7f8d7846bc20, timeout=6000) at locks/unix/thread_cond.c:89 rv = then = abstime = {tv_sec = 1491843095, tv_nsec = 700198000} #5 0x55e62c0951c5 in r_wait_space (premain=, pbl=0x7f8de853da60, block=APR_BLOCK_READ, beam=0x7f8d7846bb08) at h2_bucket_beam.c:337 status = #6 append_bucket (pbl=0x7f8de853da60, block=APR_BLOCK_READ, b=0x7f8d7846dd38, beam=0x7f8d7846bb08) at h2_bucket_beam.c:776 data = 0x7f8de853dac8 "\240\037" len = 0 space_left = 0 status = #7 h2_beam_send (beam=0x7f8d7846bb08, sender_bb=0x7f8d7864ba90, block=APR_BLOCK_READ) at h2_bucket_beam.c:906 force_report = 1 b = 0x7f8d7846dd38 status = 0 bl = {mutex = 0x7f8d7846bc20, leave = 0x55e62c094250 , leave_ctx = 0x0} #8 0x55e62c08f3aa in send_out (task=0x7f8d7846f740, bb=0x7f8d7864ba90, block=1) at h2_task.c:100 written = 8096 left = 140245585033024 status = 1 #9 0x55e62c08f976 in slave_out (task=0x7f8d7846f740, f=0x7f8d7846bdd8, bb=0x7f8d7864ba90) at h2_task.c:177 status = -512 flush = 0 blocking = 1 #10 0x55e62c08fbfd in h2_filter_slave_output (filter=0x7f8d7846bdd8, brigade=0x7f8d7864ba90) at h2_task.c:370 task = 0x7f8d7846f740 status = #11 0x55e62bffc01c in ap_content_length_filter (f=0x7f8d78472ca8, b=0x7f8d7864ba90) at protocol.c:1713 r = 0x7f8d78471750 ctx = 0x7f8d7864bc90 e = 0x7f8d7864ba98 eblock = (unknown: 2159446012) #12 0x55e62c04853d in deflate_out_filter (f=0x7f8d7864b650, bb=0x7f8d7864b8f8) at mod_deflate.c:961 rv = -512 r = 0x7f8d78471750 ctx = 0x7f8d7864b9a8 len = 8096 blen = 1157663 data = 0x7f8d7e40b000 "/*! jQuery v2.1.4 | (c) 2005, 2015 jQuery Foundation, Inc. | jquery.org/license */\n!function(a,b){\"object\"==typeof module&&\"object\"==typeof module.exports?module.exports=a.document?b(a,!0):function(a)"... c = 0x55e62c7a7498 #13 0x55e62c044b1d in filter_harness (f=0x7f8d7846bc28, bb=0x80) at mod_filter.c:323 ret = -512 cachecontrol = 0xfe00 >>> at address
Re: hanging apache httpd processes due to mod_h2
Hi Stefan, this is now with v1.10.2 but it does not help. PID old gen: 2 15608 yes (old gen) 2 no 0 0 0 0 0 requests in G state: 2-2 15608 4/36/36 G 252.75 30695 41 22.10.870.87 A.B.C.D h2 XYZ:443 GET /themes/Frontend/Responsive/frontend/_public/src/js/vendors 2-2 15608 2/5/5 G 69.62 39645 36 104.0 0.180.18 A.B.C.D h2 XYZ:443 GET /web/cache/1492526985_9e591f3c4f420ebca36f3c1be30bd5ee.css thread all bt: https://pastebin.com/raw/Cq9ymt9u I think the interesting thread is: Thread 6 Greets, Stefan Am 18.04.2017 um 15:06 schrieb Stefan Priebe - Profihost AG: > > Am 18.04.2017 um 15:03 schrieb Stefan Eissing: >> Stefan, >> >> that is a 1.10.0, right? That was the first version without nested locking >> and I fixed 2 possible dead locks in 1.10.1. >> >> I am about to release a 1.10.2 with added conformity checks and a fix for >> client omitting EOF flags. Could you give that one a try? > > Yes sure where is 1.10.2? > > Stefan > >> >> -Stefan >> >>> Am 18.04.2017 um 14:57 schrieb Stefan Priebe - Profihost AG >>>: >>> >>> Hi, >>> >>> i saw that all of them are still serving one h2 connection. >>> >>> server-status: >>> 0-3 32375 42/64/181 G 30.09 1020776 214 1285.1 2.40 >>> 5.81 >>> h081217236127.dyn.cm.kabsi.at h2 :443GET >>> /wp-content/uploads/Bloglr21-8003asd.jpg HTTP/2.0 >>> >>> And they all have Mode of operation => G >>> >>> thread apply all bt full shows: >>> #0 0x7f8e80b687fc in __lll_lock_wait () from >>> /lib/x86_64-linux-gnu/libpthread.so.0 >>> No symbol table info available. >>> #1 0x7f8e80b6b6f8 in _L_cond_lock_886 () from >>> /lib/x86_64-linux-gnu/libpthread.so.0 >>> No symbol table info available. >>> #2 0x7f8e80b6b464 in __pthread_mutex_cond_lock () from >>> /lib/x86_64-linux-gnu/libpthread.so.0 >>> No symbol table info available. >>> #3 0x7f8e80b6650a in pthread_cond_timedwait@@GLIBC_2.3.2 () from >>> /lib/x86_64-linux-gnu/libpthread.so.0 >>> No symbol table info available. >>> #4 0x7f8e80d92322 in apr_thread_cond_timedwait >>> (cond=0x7f8d7846bc70, mutex=0x7f8d7846bc20, timeout=6000) at >>> locks/unix/thread_cond.c:89 >>>rv = >>>then = >>>abstime = {tv_sec = 1491843095, tv_nsec = 700198000} >>> #5 0x55e62c0951c5 in r_wait_space (premain=, >>> pbl=0x7f8de853da60, block=APR_BLOCK_READ, beam=0x7f8d7846bb08) at >>> h2_bucket_beam.c:337 >>>status = >>> #6 append_bucket (pbl=0x7f8de853da60, block=APR_BLOCK_READ, >>> b=0x7f8d7846dd38, beam=0x7f8d7846bb08) at h2_bucket_beam.c:776 >>>data = 0x7f8de853dac8 "\240\037" >>>len = 0 >>>space_left = 0 >>>status = >>> #7 h2_beam_send (beam=0x7f8d7846bb08, sender_bb=0x7f8d7864ba90, >>> block=APR_BLOCK_READ) at h2_bucket_beam.c:906 >>>force_report = 1 >>>b = 0x7f8d7846dd38 >>>status = 0 >>>bl = {mutex = 0x7f8d7846bc20, leave = 0x55e62c094250 >>> , leave_ctx = 0x0} >>> #8 0x55e62c08f3aa in send_out (task=0x7f8d7846f740, >>> bb=0x7f8d7864ba90, block=1) at h2_task.c:100 >>>written = 8096 >>>left = 140245585033024 >>>status = 1 >>> #9 0x55e62c08f976 in slave_out (task=0x7f8d7846f740, >>> f=0x7f8d7846bdd8, bb=0x7f8d7864ba90) at h2_task.c:177 >>>status = -512 >>>flush = 0 >>>blocking = 1 >>> #10 0x55e62c08fbfd in h2_filter_slave_output (filter=0x7f8d7846bdd8, >>> brigade=0x7f8d7864ba90) at h2_task.c:370 >>>task = 0x7f8d7846f740 >>>status = >>> #11 0x55e62bffc01c in ap_content_length_filter (f=0x7f8d78472ca8, >>> b=0x7f8d7864ba90) at protocol.c:1713 >>>r = 0x7f8d78471750 >>>ctx = 0x7f8d7864bc90 >>>e = 0x7f8d7864ba98 >>>eblock = (unknown: 2159446012) >>> #12 0x55e62c04853d in deflate_out_filter (f=0x7f8d7864b650, >>> bb=0x7f8d7864b8f8) at mod_deflate.c:961 >>>rv = -512 >>>r = 0x7f8d78471750 >>>ctx = 0x7f8d7864b9a8 >>>len = 8096 >>>blen = 1157663 >>>data = 0x7f8d7e40b000 "/*! jQuery v2.1.4 | (c) 2005, 2015 jQuery >>> Foundation, Inc. | jquery.org/license >>> */\n!function(a,b){\"object\"==typeof module&&\"object\"==typeof >>> module.exports?module.exports=a.document?b(a,!0):function(a)"... >>>c = 0x55e62c7a7498 >>> #13 0x55e62c044b1d in filter_harness (f=0x7f8d7846bc28, bb=0x80) at >>> mod_filter.c:323 >>>ret = -512 >>>cachecontrol = 0xfe00 >> at address 0xfe00> >>>filter = 0xe >>> #14 0x55e62c002242 in default_handler (r=0x7f8d78471750) at core.c:4746 >>>c = 0x7f8d7846b720 >>>bb = 0x7f8d7864b8f8 >>>e = 0xfe00 >>>d = 0x7f8d7864dfa0 >>>fd = 0x7f8d7864b7b8 >>>bld_content_md5 = 2019866872 >>> #15 0x55e62c014130 in ap_run_handler
Re: hanging apache httpd processes
Here we go: https://github.com/icing/mod_h2/releases/tag/v1.10.2 > Am 18.04.2017 um 15:11 schrieb Stefan Eissing: > > In transit. Just some minutes away... > >> Am 18.04.2017 um 15:06 schrieb Stefan Priebe - Profihost AG >> : >> >> >> Am 18.04.2017 um 15:03 schrieb Stefan Eissing: >>> Stefan, >>> >>> that is a 1.10.0, right? That was the first version without nested locking >>> and I fixed 2 possible dead locks in 1.10.1. >>> >>> I am about to release a 1.10.2 with added conformity checks and a fix for >>> client omitting EOF flags. Could you give that one a try? >> >> Yes sure where is 1.10.2? >> >> Stefan >> >>> >>> -Stefan >>> Am 18.04.2017 um 14:57 schrieb Stefan Priebe - Profihost AG : Hi, i saw that all of them are still serving one h2 connection. server-status: 0-332375 42/64/181 G 30.09 1020776 214 1285.1 2.405.81 h081217236127.dyn.cm.kabsi.at h2 :443GET /wp-content/uploads/Bloglr21-8003asd.jpg HTTP/2.0 And they all have Mode of operation => G thread apply all bt full shows: #0 0x7f8e80b687fc in __lll_lock_wait () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #1 0x7f8e80b6b6f8 in _L_cond_lock_886 () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #2 0x7f8e80b6b464 in __pthread_mutex_cond_lock () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #3 0x7f8e80b6650a in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #4 0x7f8e80d92322 in apr_thread_cond_timedwait (cond=0x7f8d7846bc70, mutex=0x7f8d7846bc20, timeout=6000) at locks/unix/thread_cond.c:89 rv = then = abstime = {tv_sec = 1491843095, tv_nsec = 700198000} #5 0x55e62c0951c5 in r_wait_space (premain=, pbl=0x7f8de853da60, block=APR_BLOCK_READ, beam=0x7f8d7846bb08) at h2_bucket_beam.c:337 status = #6 append_bucket (pbl=0x7f8de853da60, block=APR_BLOCK_READ, b=0x7f8d7846dd38, beam=0x7f8d7846bb08) at h2_bucket_beam.c:776 data = 0x7f8de853dac8 "\240\037" len = 0 space_left = 0 status = #7 h2_beam_send (beam=0x7f8d7846bb08, sender_bb=0x7f8d7864ba90, block=APR_BLOCK_READ) at h2_bucket_beam.c:906 force_report = 1 b = 0x7f8d7846dd38 status = 0 bl = {mutex = 0x7f8d7846bc20, leave = 0x55e62c094250 , leave_ctx = 0x0} #8 0x55e62c08f3aa in send_out (task=0x7f8d7846f740, bb=0x7f8d7864ba90, block=1) at h2_task.c:100 written = 8096 left = 140245585033024 status = 1 #9 0x55e62c08f976 in slave_out (task=0x7f8d7846f740, f=0x7f8d7846bdd8, bb=0x7f8d7864ba90) at h2_task.c:177 status = -512 flush = 0 blocking = 1 #10 0x55e62c08fbfd in h2_filter_slave_output (filter=0x7f8d7846bdd8, brigade=0x7f8d7864ba90) at h2_task.c:370 task = 0x7f8d7846f740 status = #11 0x55e62bffc01c in ap_content_length_filter (f=0x7f8d78472ca8, b=0x7f8d7864ba90) at protocol.c:1713 r = 0x7f8d78471750 ctx = 0x7f8d7864bc90 e = 0x7f8d7864ba98 eblock = (unknown: 2159446012) #12 0x55e62c04853d in deflate_out_filter (f=0x7f8d7864b650, bb=0x7f8d7864b8f8) at mod_deflate.c:961 rv = -512 r = 0x7f8d78471750 ctx = 0x7f8d7864b9a8 len = 8096 blen = 1157663 data = 0x7f8d7e40b000 "/*! jQuery v2.1.4 | (c) 2005, 2015 jQuery Foundation, Inc. | jquery.org/license */\n!function(a,b){\"object\"==typeof module&&\"object\"==typeof module.exports?module.exports=a.document?b(a,!0):function(a)"... c = 0x55e62c7a7498 #13 0x55e62c044b1d in filter_harness (f=0x7f8d7846bc28, bb=0x80) at mod_filter.c:323 ret = -512 cachecontrol = 0xfe00 >>> at address 0xfe00> filter = 0xe #14 0x55e62c002242 in default_handler (r=0x7f8d78471750) at core.c:4746 c = 0x7f8d7846b720 bb = 0x7f8d7864b8f8 e = 0xfe00 d = 0x7f8d7864dfa0 fd = 0x7f8d7864b7b8 bld_content_md5 = 2019866872 #15 0x55e62c014130 in ap_run_handler (r=r@entry=0x7f8d78471750) at config.c:170 pHook = n = 11 rv = -1 #16 0x55e62c014679 in ap_invoke_handler (r=0x7f8d78471750) at config.c:434 handler = p = result = old_handler = 0x0 ignore = #17 0x55e62c04dcb2 in ap_process_async_request
Re: hanging apache httpd processes
In transit. Just some minutes away... > Am 18.04.2017 um 15:06 schrieb Stefan Priebe - Profihost AG >: > > > Am 18.04.2017 um 15:03 schrieb Stefan Eissing: >> Stefan, >> >> that is a 1.10.0, right? That was the first version without nested locking >> and I fixed 2 possible dead locks in 1.10.1. >> >> I am about to release a 1.10.2 with added conformity checks and a fix for >> client omitting EOF flags. Could you give that one a try? > > Yes sure where is 1.10.2? > > Stefan > >> >> -Stefan >> >>> Am 18.04.2017 um 14:57 schrieb Stefan Priebe - Profihost AG >>> : >>> >>> Hi, >>> >>> i saw that all of them are still serving one h2 connection. >>> >>> server-status: >>> 0-3 32375 42/64/181 G 30.09 1020776 214 1285.1 2.40 >>> 5.81 >>> h081217236127.dyn.cm.kabsi.at h2 :443GET >>> /wp-content/uploads/Bloglr21-8003asd.jpg HTTP/2.0 >>> >>> And they all have Mode of operation => G >>> >>> thread apply all bt full shows: >>> #0 0x7f8e80b687fc in __lll_lock_wait () from >>> /lib/x86_64-linux-gnu/libpthread.so.0 >>> No symbol table info available. >>> #1 0x7f8e80b6b6f8 in _L_cond_lock_886 () from >>> /lib/x86_64-linux-gnu/libpthread.so.0 >>> No symbol table info available. >>> #2 0x7f8e80b6b464 in __pthread_mutex_cond_lock () from >>> /lib/x86_64-linux-gnu/libpthread.so.0 >>> No symbol table info available. >>> #3 0x7f8e80b6650a in pthread_cond_timedwait@@GLIBC_2.3.2 () from >>> /lib/x86_64-linux-gnu/libpthread.so.0 >>> No symbol table info available. >>> #4 0x7f8e80d92322 in apr_thread_cond_timedwait >>> (cond=0x7f8d7846bc70, mutex=0x7f8d7846bc20, timeout=6000) at >>> locks/unix/thread_cond.c:89 >>> rv = >>> then = >>> abstime = {tv_sec = 1491843095, tv_nsec = 700198000} >>> #5 0x55e62c0951c5 in r_wait_space (premain=, >>> pbl=0x7f8de853da60, block=APR_BLOCK_READ, beam=0x7f8d7846bb08) at >>> h2_bucket_beam.c:337 >>> status = >>> #6 append_bucket (pbl=0x7f8de853da60, block=APR_BLOCK_READ, >>> b=0x7f8d7846dd38, beam=0x7f8d7846bb08) at h2_bucket_beam.c:776 >>> data = 0x7f8de853dac8 "\240\037" >>> len = 0 >>> space_left = 0 >>> status = >>> #7 h2_beam_send (beam=0x7f8d7846bb08, sender_bb=0x7f8d7864ba90, >>> block=APR_BLOCK_READ) at h2_bucket_beam.c:906 >>> force_report = 1 >>> b = 0x7f8d7846dd38 >>> status = 0 >>> bl = {mutex = 0x7f8d7846bc20, leave = 0x55e62c094250 >>> , leave_ctx = 0x0} >>> #8 0x55e62c08f3aa in send_out (task=0x7f8d7846f740, >>> bb=0x7f8d7864ba90, block=1) at h2_task.c:100 >>> written = 8096 >>> left = 140245585033024 >>> status = 1 >>> #9 0x55e62c08f976 in slave_out (task=0x7f8d7846f740, >>> f=0x7f8d7846bdd8, bb=0x7f8d7864ba90) at h2_task.c:177 >>> status = -512 >>> flush = 0 >>> blocking = 1 >>> #10 0x55e62c08fbfd in h2_filter_slave_output (filter=0x7f8d7846bdd8, >>> brigade=0x7f8d7864ba90) at h2_task.c:370 >>> task = 0x7f8d7846f740 >>> status = >>> #11 0x55e62bffc01c in ap_content_length_filter (f=0x7f8d78472ca8, >>> b=0x7f8d7864ba90) at protocol.c:1713 >>> r = 0x7f8d78471750 >>> ctx = 0x7f8d7864bc90 >>> e = 0x7f8d7864ba98 >>> eblock = (unknown: 2159446012) >>> #12 0x55e62c04853d in deflate_out_filter (f=0x7f8d7864b650, >>> bb=0x7f8d7864b8f8) at mod_deflate.c:961 >>> rv = -512 >>> r = 0x7f8d78471750 >>> ctx = 0x7f8d7864b9a8 >>> len = 8096 >>> blen = 1157663 >>> data = 0x7f8d7e40b000 "/*! jQuery v2.1.4 | (c) 2005, 2015 jQuery >>> Foundation, Inc. | jquery.org/license >>> */\n!function(a,b){\"object\"==typeof module&&\"object\"==typeof >>> module.exports?module.exports=a.document?b(a,!0):function(a)"... >>> c = 0x55e62c7a7498 >>> #13 0x55e62c044b1d in filter_harness (f=0x7f8d7846bc28, bb=0x80) at >>> mod_filter.c:323 >>> ret = -512 >>> cachecontrol = 0xfe00 >> at address 0xfe00> >>> filter = 0xe >>> #14 0x55e62c002242 in default_handler (r=0x7f8d78471750) at core.c:4746 >>> c = 0x7f8d7846b720 >>> bb = 0x7f8d7864b8f8 >>> e = 0xfe00 >>> d = 0x7f8d7864dfa0 >>> fd = 0x7f8d7864b7b8 >>> bld_content_md5 = 2019866872 >>> #15 0x55e62c014130 in ap_run_handler (r=r@entry=0x7f8d78471750) at >>> config.c:170 >>> pHook = >>> n = 11 >>> rv = -1 >>> #16 0x55e62c014679 in ap_invoke_handler (r=0x7f8d78471750) at >>> config.c:434 >>> handler = >>> p = >>> result = >>> old_handler = 0x0 >>> ignore = >>> #17 0x55e62c04dcb2 in ap_process_async_request (r=0x7f8d78471750) at >>> http_request.c:436 >>> c = 0x7f8d7846b720 >>> access_status = 0 >>> #18 0x55e62c04de50 in ap_process_request (r=0x7f8d78471750) at >>> http_request.c:471 >>> bb = 0x7f8d7846bd80 >>> c =
Re: hanging apache httpd processes
Am 18.04.2017 um 15:03 schrieb Stefan Eissing: > Stefan, > > that is a 1.10.0, right? That was the first version without nested locking > and I fixed 2 possible dead locks in 1.10.1. > > I am about to release a 1.10.2 with added conformity checks and a fix for > client omitting EOF flags. Could you give that one a try? Yes sure where is 1.10.2? Stefan > > -Stefan > >> Am 18.04.2017 um 14:57 schrieb Stefan Priebe - Profihost AG >>: >> >> Hi, >> >> i saw that all of them are still serving one h2 connection. >> >> server-status: >> 0-3 32375 42/64/181 G 30.09 1020776 214 1285.1 2.40 >> 5.81 >> h081217236127.dyn.cm.kabsi.ath2 :443GET >> /wp-content/uploads/Bloglr21-8003asd.jpg HTTP/2.0 >> >> And they all have Mode of operation => G >> >> thread apply all bt full shows: >> #0 0x7f8e80b687fc in __lll_lock_wait () from >> /lib/x86_64-linux-gnu/libpthread.so.0 >> No symbol table info available. >> #1 0x7f8e80b6b6f8 in _L_cond_lock_886 () from >> /lib/x86_64-linux-gnu/libpthread.so.0 >> No symbol table info available. >> #2 0x7f8e80b6b464 in __pthread_mutex_cond_lock () from >> /lib/x86_64-linux-gnu/libpthread.so.0 >> No symbol table info available. >> #3 0x7f8e80b6650a in pthread_cond_timedwait@@GLIBC_2.3.2 () from >> /lib/x86_64-linux-gnu/libpthread.so.0 >> No symbol table info available. >> #4 0x7f8e80d92322 in apr_thread_cond_timedwait >> (cond=0x7f8d7846bc70, mutex=0x7f8d7846bc20, timeout=6000) at >> locks/unix/thread_cond.c:89 >>rv = >>then = >>abstime = {tv_sec = 1491843095, tv_nsec = 700198000} >> #5 0x55e62c0951c5 in r_wait_space (premain=, >> pbl=0x7f8de853da60, block=APR_BLOCK_READ, beam=0x7f8d7846bb08) at >> h2_bucket_beam.c:337 >>status = >> #6 append_bucket (pbl=0x7f8de853da60, block=APR_BLOCK_READ, >> b=0x7f8d7846dd38, beam=0x7f8d7846bb08) at h2_bucket_beam.c:776 >>data = 0x7f8de853dac8 "\240\037" >>len = 0 >>space_left = 0 >>status = >> #7 h2_beam_send (beam=0x7f8d7846bb08, sender_bb=0x7f8d7864ba90, >> block=APR_BLOCK_READ) at h2_bucket_beam.c:906 >>force_report = 1 >>b = 0x7f8d7846dd38 >>status = 0 >>bl = {mutex = 0x7f8d7846bc20, leave = 0x55e62c094250 >> , leave_ctx = 0x0} >> #8 0x55e62c08f3aa in send_out (task=0x7f8d7846f740, >> bb=0x7f8d7864ba90, block=1) at h2_task.c:100 >>written = 8096 >>left = 140245585033024 >>status = 1 >> #9 0x55e62c08f976 in slave_out (task=0x7f8d7846f740, >> f=0x7f8d7846bdd8, bb=0x7f8d7864ba90) at h2_task.c:177 >>status = -512 >>flush = 0 >>blocking = 1 >> #10 0x55e62c08fbfd in h2_filter_slave_output (filter=0x7f8d7846bdd8, >> brigade=0x7f8d7864ba90) at h2_task.c:370 >>task = 0x7f8d7846f740 >>status = >> #11 0x55e62bffc01c in ap_content_length_filter (f=0x7f8d78472ca8, >> b=0x7f8d7864ba90) at protocol.c:1713 >>r = 0x7f8d78471750 >>ctx = 0x7f8d7864bc90 >>e = 0x7f8d7864ba98 >>eblock = (unknown: 2159446012) >> #12 0x55e62c04853d in deflate_out_filter (f=0x7f8d7864b650, >> bb=0x7f8d7864b8f8) at mod_deflate.c:961 >>rv = -512 >>r = 0x7f8d78471750 >>ctx = 0x7f8d7864b9a8 >>len = 8096 >>blen = 1157663 >>data = 0x7f8d7e40b000 "/*! jQuery v2.1.4 | (c) 2005, 2015 jQuery >> Foundation, Inc. | jquery.org/license >> */\n!function(a,b){\"object\"==typeof module&&\"object\"==typeof >> module.exports?module.exports=a.document?b(a,!0):function(a)"... >>c = 0x55e62c7a7498 >> #13 0x55e62c044b1d in filter_harness (f=0x7f8d7846bc28, bb=0x80) at >> mod_filter.c:323 >>ret = -512 >>cachecontrol = 0xfe00 > at address 0xfe00> >>filter = 0xe >> #14 0x55e62c002242 in default_handler (r=0x7f8d78471750) at core.c:4746 >>c = 0x7f8d7846b720 >>bb = 0x7f8d7864b8f8 >>e = 0xfe00 >>d = 0x7f8d7864dfa0 >>fd = 0x7f8d7864b7b8 >>bld_content_md5 = 2019866872 >> #15 0x55e62c014130 in ap_run_handler (r=r@entry=0x7f8d78471750) at >> config.c:170 >>pHook = >>n = 11 >>rv = -1 >> #16 0x55e62c014679 in ap_invoke_handler (r=0x7f8d78471750) at >> config.c:434 >>handler = >>p = >>result = >>old_handler = 0x0 >>ignore = >> #17 0x55e62c04dcb2 in ap_process_async_request (r=0x7f8d78471750) at >> http_request.c:436 >>c = 0x7f8d7846b720 >>access_status = 0 >> #18 0x55e62c04de50 in ap_process_request (r=0x7f8d78471750) at >> http_request.c:471 >>bb = 0x7f8d7846bd80 >>c = 0x7f8d7846b720 >>rv = -512 >> #19 0x55e62c08ec32 in h2_task_process_request (c=, >> task=) at h2_task.c:666 >>req = 0x7f8d78471750 >>cs = 0x7f8d7846bd80 >>r = 0x7f8d78471750 >> #20
Re: hanging apache httpd processes
Stefan, that is a 1.10.0, right? That was the first version without nested locking and I fixed 2 possible dead locks in 1.10.1. I am about to release a 1.10.2 with added conformity checks and a fix for client omitting EOF flags. Could you give that one a try? -Stefan > Am 18.04.2017 um 14:57 schrieb Stefan Priebe - Profihost AG >: > > Hi, > > i saw that all of them are still serving one h2 connection. > > server-status: > 0-3 32375 42/64/181 G 30.09 1020776 214 1285.1 2.40 > 5.81 > h081217236127.dyn.cm.kabsi.at h2 :443GET > /wp-content/uploads/Bloglr21-8003asd.jpg HTTP/2.0 > > And they all have Mode of operation => G > > thread apply all bt full shows: > #0 0x7f8e80b687fc in __lll_lock_wait () from > /lib/x86_64-linux-gnu/libpthread.so.0 > No symbol table info available. > #1 0x7f8e80b6b6f8 in _L_cond_lock_886 () from > /lib/x86_64-linux-gnu/libpthread.so.0 > No symbol table info available. > #2 0x7f8e80b6b464 in __pthread_mutex_cond_lock () from > /lib/x86_64-linux-gnu/libpthread.so.0 > No symbol table info available. > #3 0x7f8e80b6650a in pthread_cond_timedwait@@GLIBC_2.3.2 () from > /lib/x86_64-linux-gnu/libpthread.so.0 > No symbol table info available. > #4 0x7f8e80d92322 in apr_thread_cond_timedwait > (cond=0x7f8d7846bc70, mutex=0x7f8d7846bc20, timeout=6000) at > locks/unix/thread_cond.c:89 >rv = >then = >abstime = {tv_sec = 1491843095, tv_nsec = 700198000} > #5 0x55e62c0951c5 in r_wait_space (premain=, > pbl=0x7f8de853da60, block=APR_BLOCK_READ, beam=0x7f8d7846bb08) at > h2_bucket_beam.c:337 >status = > #6 append_bucket (pbl=0x7f8de853da60, block=APR_BLOCK_READ, > b=0x7f8d7846dd38, beam=0x7f8d7846bb08) at h2_bucket_beam.c:776 >data = 0x7f8de853dac8 "\240\037" >len = 0 >space_left = 0 >status = > #7 h2_beam_send (beam=0x7f8d7846bb08, sender_bb=0x7f8d7864ba90, > block=APR_BLOCK_READ) at h2_bucket_beam.c:906 >force_report = 1 >b = 0x7f8d7846dd38 >status = 0 >bl = {mutex = 0x7f8d7846bc20, leave = 0x55e62c094250 > , leave_ctx = 0x0} > #8 0x55e62c08f3aa in send_out (task=0x7f8d7846f740, > bb=0x7f8d7864ba90, block=1) at h2_task.c:100 >written = 8096 >left = 140245585033024 >status = 1 > #9 0x55e62c08f976 in slave_out (task=0x7f8d7846f740, > f=0x7f8d7846bdd8, bb=0x7f8d7864ba90) at h2_task.c:177 >status = -512 >flush = 0 >blocking = 1 > #10 0x55e62c08fbfd in h2_filter_slave_output (filter=0x7f8d7846bdd8, > brigade=0x7f8d7864ba90) at h2_task.c:370 >task = 0x7f8d7846f740 >status = > #11 0x55e62bffc01c in ap_content_length_filter (f=0x7f8d78472ca8, > b=0x7f8d7864ba90) at protocol.c:1713 >r = 0x7f8d78471750 >ctx = 0x7f8d7864bc90 >e = 0x7f8d7864ba98 >eblock = (unknown: 2159446012) > #12 0x55e62c04853d in deflate_out_filter (f=0x7f8d7864b650, > bb=0x7f8d7864b8f8) at mod_deflate.c:961 >rv = -512 >r = 0x7f8d78471750 >ctx = 0x7f8d7864b9a8 >len = 8096 >blen = 1157663 >data = 0x7f8d7e40b000 "/*! jQuery v2.1.4 | (c) 2005, 2015 jQuery > Foundation, Inc. | jquery.org/license > */\n!function(a,b){\"object\"==typeof module&&\"object\"==typeof > module.exports?module.exports=a.document?b(a,!0):function(a)"... >c = 0x55e62c7a7498 > #13 0x55e62c044b1d in filter_harness (f=0x7f8d7846bc28, bb=0x80) at > mod_filter.c:323 >ret = -512 >cachecontrol = 0xfe00 at address 0xfe00> >filter = 0xe > #14 0x55e62c002242 in default_handler (r=0x7f8d78471750) at core.c:4746 >c = 0x7f8d7846b720 >bb = 0x7f8d7864b8f8 >e = 0xfe00 >d = 0x7f8d7864dfa0 >fd = 0x7f8d7864b7b8 >bld_content_md5 = 2019866872 > #15 0x55e62c014130 in ap_run_handler (r=r@entry=0x7f8d78471750) at > config.c:170 >pHook = >n = 11 >rv = -1 > #16 0x55e62c014679 in ap_invoke_handler (r=0x7f8d78471750) at > config.c:434 >handler = >p = >result = >old_handler = 0x0 >ignore = > #17 0x55e62c04dcb2 in ap_process_async_request (r=0x7f8d78471750) at > http_request.c:436 >c = 0x7f8d7846b720 >access_status = 0 > #18 0x55e62c04de50 in ap_process_request (r=0x7f8d78471750) at > http_request.c:471 >bb = 0x7f8d7846bd80 >c = 0x7f8d7846b720 >rv = -512 > #19 0x55e62c08ec32 in h2_task_process_request (c=, > task=) at h2_task.c:666 >req = 0x7f8d78471750 >cs = 0x7f8d7846bd80 >r = 0x7f8d78471750 > #20 h2_task_process_conn (c=0x7f8d7846bc28) at h2_task.c:713 >ctx = 0x7f8d7846bd80 > #21 0x55e62c01e130 in ap_run_process_connection (c=0x7f8d7846b720) > at connection.c:42 >pHook = >n = 0 >rv = -1 > #22
Re: hanging apache httpd processes
Hi, i saw that all of them are still serving one h2 connection. server-status: 0-3 32375 42/64/181 G 30.09 1020776 214 1285.1 2.40 5.81 h081217236127.dyn.cm.kabsi.at h2 :443GET /wp-content/uploads/Bloglr21-8003asd.jpg HTTP/2.0 And they all have Mode of operation => G thread apply all bt full shows: #0 0x7f8e80b687fc in __lll_lock_wait () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #1 0x7f8e80b6b6f8 in _L_cond_lock_886 () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #2 0x7f8e80b6b464 in __pthread_mutex_cond_lock () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #3 0x7f8e80b6650a in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #4 0x7f8e80d92322 in apr_thread_cond_timedwait (cond=0x7f8d7846bc70, mutex=0x7f8d7846bc20, timeout=6000) at locks/unix/thread_cond.c:89 rv = then = abstime = {tv_sec = 1491843095, tv_nsec = 700198000} #5 0x55e62c0951c5 in r_wait_space (premain=, pbl=0x7f8de853da60, block=APR_BLOCK_READ, beam=0x7f8d7846bb08) at h2_bucket_beam.c:337 status = #6 append_bucket (pbl=0x7f8de853da60, block=APR_BLOCK_READ, b=0x7f8d7846dd38, beam=0x7f8d7846bb08) at h2_bucket_beam.c:776 data = 0x7f8de853dac8 "\240\037" len = 0 space_left = 0 status = #7 h2_beam_send (beam=0x7f8d7846bb08, sender_bb=0x7f8d7864ba90, block=APR_BLOCK_READ) at h2_bucket_beam.c:906 force_report = 1 b = 0x7f8d7846dd38 status = 0 bl = {mutex = 0x7f8d7846bc20, leave = 0x55e62c094250 , leave_ctx = 0x0} #8 0x55e62c08f3aa in send_out (task=0x7f8d7846f740, bb=0x7f8d7864ba90, block=1) at h2_task.c:100 written = 8096 left = 140245585033024 status = 1 #9 0x55e62c08f976 in slave_out (task=0x7f8d7846f740, f=0x7f8d7846bdd8, bb=0x7f8d7864ba90) at h2_task.c:177 status = -512 flush = 0 blocking = 1 #10 0x55e62c08fbfd in h2_filter_slave_output (filter=0x7f8d7846bdd8, brigade=0x7f8d7864ba90) at h2_task.c:370 task = 0x7f8d7846f740 status = #11 0x55e62bffc01c in ap_content_length_filter (f=0x7f8d78472ca8, b=0x7f8d7864ba90) at protocol.c:1713 r = 0x7f8d78471750 ctx = 0x7f8d7864bc90 e = 0x7f8d7864ba98 eblock = (unknown: 2159446012) #12 0x55e62c04853d in deflate_out_filter (f=0x7f8d7864b650, bb=0x7f8d7864b8f8) at mod_deflate.c:961 rv = -512 r = 0x7f8d78471750 ctx = 0x7f8d7864b9a8 len = 8096 blen = 1157663 data = 0x7f8d7e40b000 "/*! jQuery v2.1.4 | (c) 2005, 2015 jQuery Foundation, Inc. | jquery.org/license */\n!function(a,b){\"object\"==typeof module&&\"object\"==typeof module.exports?module.exports=a.document?b(a,!0):function(a)"... c = 0x55e62c7a7498 #13 0x55e62c044b1d in filter_harness (f=0x7f8d7846bc28, bb=0x80) at mod_filter.c:323 ret = -512 cachecontrol = 0xfe00 filter = 0xe #14 0x55e62c002242 in default_handler (r=0x7f8d78471750) at core.c:4746 c = 0x7f8d7846b720 bb = 0x7f8d7864b8f8 e = 0xfe00 d = 0x7f8d7864dfa0 fd = 0x7f8d7864b7b8 bld_content_md5 = 2019866872 #15 0x55e62c014130 in ap_run_handler (r=r@entry=0x7f8d78471750) at config.c:170 pHook = n = 11 rv = -1 #16 0x55e62c014679 in ap_invoke_handler (r=0x7f8d78471750) at config.c:434 handler = p = result = old_handler = 0x0 ignore = #17 0x55e62c04dcb2 in ap_process_async_request (r=0x7f8d78471750) at http_request.c:436 c = 0x7f8d7846b720 access_status = 0 #18 0x55e62c04de50 in ap_process_request (r=0x7f8d78471750) at http_request.c:471 bb = 0x7f8d7846bd80 c = 0x7f8d7846b720 rv = -512 #19 0x55e62c08ec32 in h2_task_process_request (c=, task=) at h2_task.c:666 req = 0x7f8d78471750 cs = 0x7f8d7846bd80 r = 0x7f8d78471750 #20 h2_task_process_conn (c=0x7f8d7846bc28) at h2_task.c:713 ctx = 0x7f8d7846bd80 #21 0x55e62c01e130 in ap_run_process_connection (c=0x7f8d7846b720) at connection.c:42 pHook = n = 0 rv = -1 #22 0x55e62c09012b in h2_task_do (task=0x7f8d7846f740, thread=0x55e62cb0ce70, worker_id=0) at h2_task.c:623 c = 0x7f8d7846b720 #23 0x55e62c093619 in slot_run (thread=0x55e62cb0ce70, wctx=0x55e62c8b9268) at h2_workers.c:217 slot = 0x55e62c8b9268 #24 0x7f8e80b620a4 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #25 0x7f8e8089762d in clone () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. Thread 3 (Thread 0x7f8da678c700 (LWP 28251)): #0 0x7f8e80b687fc in __lll_lock_wait () from
Re: hanging apache httpd processes
What Eric said. With the changes in http2 worker scheduling, if I introduced a bug in child exit there, I'd expect it to trigger via workers_pool_cleanup() in h2_workers.c -Stefan > Am 18.04.2017 um 14:47 schrieb Eric Covener: > > On Tue, Apr 18, 2017 at 8:43 AM, Stefan Priebe - Profihost AG > wrote: >> bt of such a process shows: >> (gdb) bt >> #0 0x7f5df74f64db in pthread_join () from >> /lib/x86_64-linux-gnu/libpthread.so.0 > > Need to see the other threads in the process, this one is just waiting > for the others to complete. > > "thread apply all bt full" > > > -- > Eric Covener > cove...@gmail.com
Re: hanging apache httpd processes
On Tue, Apr 18, 2017 at 8:43 AM, Stefan Priebe - Profihost AGwrote: > bt of such a process shows: > (gdb) bt > #0 0x7f5df74f64db in pthread_join () from > /lib/x86_64-linux-gnu/libpthread.so.0 Need to see the other threads in the process, this one is just waiting for the others to complete. "thread apply all bt full" -- Eric Covener cove...@gmail.com