Re: hanging apache httpd processes due to mod_h2

[Stefan Priebe - Profihost AG]

looks good so war. Will wait some more days.

Greets,
Stefan

Am 19.04.2017 um 11:09 schrieb Stefan Eissing:
> Thanks, the backtraces really helped. Could you try the following patch?
> 
> -Stefan
> 
> 
> 
> 
>> Am 19.04.2017 um 08:32 schrieb Stefan Priebe - Profihost AG 
>> :
>>
>> Hi Stefan,
>>
>> this is now with v1.10.2 but it does not help.
>>
>> PID old gen:
>>
>> 215608   yes (old gen)   2   no  0   0   0   0   >> 0
>>
>>
>> requests in G state:
>>
>> 2-2  15608   4/36/36 G   252.75  30695   41  22.10.870.87
>> A.B.C.D h2  XYZ:443
>> GET /themes/Frontend/Responsive/frontend/_public/src/js/vendors
>>
>> 2-2  15608   2/5/5   G   69.62   39645   36  104.0   0.180.18
>> A.B.C.D h2  XYZ:443 GET
>> /web/cache/1492526985_9e591f3c4f420ebca36f3c1be30bd5ee.css
>>
>> thread all bt:
>> https://pastebin.com/raw/Cq9ymt9u
>>
>> I think the interesting thread is:
>> Thread 6
>>
>> Greets,
>> Stefan
>>
>> Am 18.04.2017 um 15:06 schrieb Stefan Priebe - Profihost AG:
>>>
>>> Am 18.04.2017 um 15:03 schrieb Stefan Eissing:
 Stefan,

 that is a 1.10.0, right? That was the first version without nested locking 
 and I fixed 2 possible dead locks in 1.10.1. 

 I am about to release a 1.10.2 with added conformity checks and a fix for 
 client omitting EOF flags. Could you give that one a try?
>>>
>>> Yes sure where is 1.10.2?
>>>
>>> Stefan
>>>

 -Stefan

> Am 18.04.2017 um 14:57 schrieb Stefan Priebe - Profihost AG 
> :
>
> Hi,
>
> i saw that all of them are still serving one h2 connection.
>
> server-status:
> 0-3   32375   42/64/181   G   30.09   1020776 214 1285.1  
> 2.405.81
> h081217236127.dyn.cm.kabsi.at h2  :443GET
> /wp-content/uploads/Bloglr21-8003asd.jpg HTTP/2.0
>
> And they all have Mode of operation => G
>
> thread apply all bt full shows:
> #0  0x7f8e80b687fc in __lll_lock_wait () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> No symbol table info available.
> #1  0x7f8e80b6b6f8 in _L_cond_lock_886 () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> No symbol table info available.
> #2  0x7f8e80b6b464 in __pthread_mutex_cond_lock () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> No symbol table info available.
> #3  0x7f8e80b6650a in pthread_cond_timedwait@@GLIBC_2.3.2 () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> No symbol table info available.
> #4  0x7f8e80d92322 in apr_thread_cond_timedwait
> (cond=0x7f8d7846bc70, mutex=0x7f8d7846bc20, timeout=6000) at
> locks/unix/thread_cond.c:89
>   rv = 
>   then = 
>   abstime = {tv_sec = 1491843095, tv_nsec = 700198000}
> #5  0x55e62c0951c5 in r_wait_space (premain=,
> pbl=0x7f8de853da60, block=APR_BLOCK_READ, beam=0x7f8d7846bb08) at
> h2_bucket_beam.c:337
>   status = 
> #6  append_bucket (pbl=0x7f8de853da60, block=APR_BLOCK_READ,
> b=0x7f8d7846dd38, beam=0x7f8d7846bb08) at h2_bucket_beam.c:776
>   data = 0x7f8de853dac8 "\240\037"
>   len = 0
>   space_left = 0
>   status = 
> #7  h2_beam_send (beam=0x7f8d7846bb08, sender_bb=0x7f8d7864ba90,
> block=APR_BLOCK_READ) at h2_bucket_beam.c:906
>   force_report = 1
>   b = 0x7f8d7846dd38
>   status = 0
>   bl = {mutex = 0x7f8d7846bc20, leave = 0x55e62c094250
> , leave_ctx = 0x0}
> #8  0x55e62c08f3aa in send_out (task=0x7f8d7846f740,
> bb=0x7f8d7864ba90, block=1) at h2_task.c:100
>   written = 8096
>   left = 140245585033024
>   status = 1
> #9  0x55e62c08f976 in slave_out (task=0x7f8d7846f740,
> f=0x7f8d7846bdd8, bb=0x7f8d7864ba90) at h2_task.c:177
>   status = -512
>   flush = 0
>   blocking = 1
> #10 0x55e62c08fbfd in h2_filter_slave_output (filter=0x7f8d7846bdd8,
> brigade=0x7f8d7864ba90) at h2_task.c:370
>   task = 0x7f8d7846f740
>   status = 
> #11 0x55e62bffc01c in ap_content_length_filter (f=0x7f8d78472ca8,
> b=0x7f8d7864ba90) at protocol.c:1713
>   r = 0x7f8d78471750
>   ctx = 0x7f8d7864bc90
>   e = 0x7f8d7864ba98
>   eblock = (unknown: 2159446012)
> #12 0x55e62c04853d in deflate_out_filter (f=0x7f8d7864b650,
> bb=0x7f8d7864b8f8) at mod_deflate.c:961
>   rv = -512
>   r = 0x7f8d78471750
>   ctx = 0x7f8d7864b9a8
>   len = 8096
>   blen = 1157663
>   data = 0x7f8d7e40b000 "/*! jQuery v2.1.4 | (c) 2005, 2015 jQuery
> Foundation, Inc. | jquery.org/license
> */\n!function(a,b){\"object\"==typeof module&&\"object\"==typeof
> module.exports?module.exports=a.document?b(a,!0):function(a)"...
>   c = 0x55e62c7a7498
> #13 

Re: hanging apache httpd processes due to mod_h2

[Stefan Eissing]

Thanks, the backtraces really helped. Could you try the following patch?

-Stefan



h2_beam_locks_v1.diff
Description: Binary data

> Am 19.04.2017 um 08:32 schrieb Stefan Priebe - Profihost AG 
> :
> 
> Hi Stefan,
> 
> this is now with v1.10.2 but it does not help.
> 
> PID old gen:
> 
> 2 15608   yes (old gen)   2   no  0   0   0   0   > 0
> 
> 
> requests in G state:
> 
> 2-2   15608   4/36/36 G   252.75  30695   41  22.10.870.87
> A.B.C.D h2  XYZ:443
> GET /themes/Frontend/Responsive/frontend/_public/src/js/vendors
> 
> 2-2   15608   2/5/5   G   69.62   39645   36  104.0   0.180.18
> A.B.C.D h2  XYZ:443 GET
> /web/cache/1492526985_9e591f3c4f420ebca36f3c1be30bd5ee.css
> 
> thread all bt:
> https://pastebin.com/raw/Cq9ymt9u
> 
> I think the interesting thread is:
> Thread 6
> 
> Greets,
> Stefan
> 
> Am 18.04.2017 um 15:06 schrieb Stefan Priebe - Profihost AG:
>> 
>> Am 18.04.2017 um 15:03 schrieb Stefan Eissing:
>>> Stefan,
>>> 
>>> that is a 1.10.0, right? That was the first version without nested locking 
>>> and I fixed 2 possible dead locks in 1.10.1. 
>>> 
>>> I am about to release a 1.10.2 with added conformity checks and a fix for 
>>> client omitting EOF flags. Could you give that one a try?
>> 
>> Yes sure where is 1.10.2?
>> 
>> Stefan
>> 
>>> 
>>> -Stefan
>>> 
 Am 18.04.2017 um 14:57 schrieb Stefan Priebe - Profihost AG 
 :
 
 Hi,
 
 i saw that all of them are still serving one h2 connection.
 
 server-status:
 0-332375   42/64/181   G   30.09   1020776 214 1285.1  
 2.405.81
 h081217236127.dyn.cm.kabsi.at  h2  :443GET
 /wp-content/uploads/Bloglr21-8003asd.jpg HTTP/2.0
 
 And they all have Mode of operation => G
 
 thread apply all bt full shows:
 #0  0x7f8e80b687fc in __lll_lock_wait () from
 /lib/x86_64-linux-gnu/libpthread.so.0
 No symbol table info available.
 #1  0x7f8e80b6b6f8 in _L_cond_lock_886 () from
 /lib/x86_64-linux-gnu/libpthread.so.0
 No symbol table info available.
 #2  0x7f8e80b6b464 in __pthread_mutex_cond_lock () from
 /lib/x86_64-linux-gnu/libpthread.so.0
 No symbol table info available.
 #3  0x7f8e80b6650a in pthread_cond_timedwait@@GLIBC_2.3.2 () from
 /lib/x86_64-linux-gnu/libpthread.so.0
 No symbol table info available.
 #4  0x7f8e80d92322 in apr_thread_cond_timedwait
 (cond=0x7f8d7846bc70, mutex=0x7f8d7846bc20, timeout=6000) at
 locks/unix/thread_cond.c:89
   rv = 
   then = 
   abstime = {tv_sec = 1491843095, tv_nsec = 700198000}
 #5  0x55e62c0951c5 in r_wait_space (premain=,
 pbl=0x7f8de853da60, block=APR_BLOCK_READ, beam=0x7f8d7846bb08) at
 h2_bucket_beam.c:337
   status = 
 #6  append_bucket (pbl=0x7f8de853da60, block=APR_BLOCK_READ,
 b=0x7f8d7846dd38, beam=0x7f8d7846bb08) at h2_bucket_beam.c:776
   data = 0x7f8de853dac8 "\240\037"
   len = 0
   space_left = 0
   status = 
 #7  h2_beam_send (beam=0x7f8d7846bb08, sender_bb=0x7f8d7864ba90,
 block=APR_BLOCK_READ) at h2_bucket_beam.c:906
   force_report = 1
   b = 0x7f8d7846dd38
   status = 0
   bl = {mutex = 0x7f8d7846bc20, leave = 0x55e62c094250
 , leave_ctx = 0x0}
 #8  0x55e62c08f3aa in send_out (task=0x7f8d7846f740,
 bb=0x7f8d7864ba90, block=1) at h2_task.c:100
   written = 8096
   left = 140245585033024
   status = 1
 #9  0x55e62c08f976 in slave_out (task=0x7f8d7846f740,
 f=0x7f8d7846bdd8, bb=0x7f8d7864ba90) at h2_task.c:177
   status = -512
   flush = 0
   blocking = 1
 #10 0x55e62c08fbfd in h2_filter_slave_output (filter=0x7f8d7846bdd8,
 brigade=0x7f8d7864ba90) at h2_task.c:370
   task = 0x7f8d7846f740
   status = 
 #11 0x55e62bffc01c in ap_content_length_filter (f=0x7f8d78472ca8,
 b=0x7f8d7864ba90) at protocol.c:1713
   r = 0x7f8d78471750
   ctx = 0x7f8d7864bc90
   e = 0x7f8d7864ba98
   eblock = (unknown: 2159446012)
 #12 0x55e62c04853d in deflate_out_filter (f=0x7f8d7864b650,
 bb=0x7f8d7864b8f8) at mod_deflate.c:961
   rv = -512
   r = 0x7f8d78471750
   ctx = 0x7f8d7864b9a8
   len = 8096
   blen = 1157663
   data = 0x7f8d7e40b000 "/*! jQuery v2.1.4 | (c) 2005, 2015 jQuery
 Foundation, Inc. | jquery.org/license
 */\n!function(a,b){\"object\"==typeof module&&\"object\"==typeof
 module.exports?module.exports=a.document?b(a,!0):function(a)"...
   c = 0x55e62c7a7498
 #13 0x55e62c044b1d in filter_harness (f=0x7f8d7846bc28, bb=0x80) at
 mod_filter.c:323
   ret = -512
   cachecontrol = 0xfe00 >>> at address 

Re: hanging apache httpd processes due to mod_h2

[Stefan Priebe - Profihost AG]

Hi Stefan,

this is now with v1.10.2 but it does not help.

PID old gen:

2   15608   yes (old gen)   2   no  0   0   0   0   0


requests in G state:

2-2 15608   4/36/36 G   252.75  30695   41  22.10.870.87
A.B.C.D h2  XYZ:443
GET /themes/Frontend/Responsive/frontend/_public/src/js/vendors

2-2 15608   2/5/5   G   69.62   39645   36  104.0   0.180.18
A.B.C.D h2  XYZ:443 GET
/web/cache/1492526985_9e591f3c4f420ebca36f3c1be30bd5ee.css

thread all bt:
https://pastebin.com/raw/Cq9ymt9u

I think the interesting thread is:
Thread 6

Greets,
Stefan

Am 18.04.2017 um 15:06 schrieb Stefan Priebe - Profihost AG:
> 
> Am 18.04.2017 um 15:03 schrieb Stefan Eissing:
>> Stefan,
>>
>> that is a 1.10.0, right? That was the first version without nested locking 
>> and I fixed 2 possible dead locks in 1.10.1. 
>>
>> I am about to release a 1.10.2 with added conformity checks and a fix for 
>> client omitting EOF flags. Could you give that one a try?
> 
> Yes sure where is 1.10.2?
> 
> Stefan
> 
>>
>> -Stefan
>>
>>> Am 18.04.2017 um 14:57 schrieb Stefan Priebe - Profihost AG 
>>> :
>>>
>>> Hi,
>>>
>>> i saw that all of them are still serving one h2 connection.
>>>
>>> server-status:
>>> 0-3 32375   42/64/181   G   30.09   1020776 214 1285.1  2.40
>>> 5.81
>>> h081217236127.dyn.cm.kabsi.at   h2  :443GET
>>> /wp-content/uploads/Bloglr21-8003asd.jpg HTTP/2.0
>>>
>>> And they all have Mode of operation => G
>>>
>>> thread apply all bt full shows:
>>> #0  0x7f8e80b687fc in __lll_lock_wait () from
>>> /lib/x86_64-linux-gnu/libpthread.so.0
>>> No symbol table info available.
>>> #1  0x7f8e80b6b6f8 in _L_cond_lock_886 () from
>>> /lib/x86_64-linux-gnu/libpthread.so.0
>>> No symbol table info available.
>>> #2  0x7f8e80b6b464 in __pthread_mutex_cond_lock () from
>>> /lib/x86_64-linux-gnu/libpthread.so.0
>>> No symbol table info available.
>>> #3  0x7f8e80b6650a in pthread_cond_timedwait@@GLIBC_2.3.2 () from
>>> /lib/x86_64-linux-gnu/libpthread.so.0
>>> No symbol table info available.
>>> #4  0x7f8e80d92322 in apr_thread_cond_timedwait
>>> (cond=0x7f8d7846bc70, mutex=0x7f8d7846bc20, timeout=6000) at
>>> locks/unix/thread_cond.c:89
>>>rv = 
>>>then = 
>>>abstime = {tv_sec = 1491843095, tv_nsec = 700198000}
>>> #5  0x55e62c0951c5 in r_wait_space (premain=,
>>> pbl=0x7f8de853da60, block=APR_BLOCK_READ, beam=0x7f8d7846bb08) at
>>> h2_bucket_beam.c:337
>>>status = 
>>> #6  append_bucket (pbl=0x7f8de853da60, block=APR_BLOCK_READ,
>>> b=0x7f8d7846dd38, beam=0x7f8d7846bb08) at h2_bucket_beam.c:776
>>>data = 0x7f8de853dac8 "\240\037"
>>>len = 0
>>>space_left = 0
>>>status = 
>>> #7  h2_beam_send (beam=0x7f8d7846bb08, sender_bb=0x7f8d7864ba90,
>>> block=APR_BLOCK_READ) at h2_bucket_beam.c:906
>>>force_report = 1
>>>b = 0x7f8d7846dd38
>>>status = 0
>>>bl = {mutex = 0x7f8d7846bc20, leave = 0x55e62c094250
>>> , leave_ctx = 0x0}
>>> #8  0x55e62c08f3aa in send_out (task=0x7f8d7846f740,
>>> bb=0x7f8d7864ba90, block=1) at h2_task.c:100
>>>written = 8096
>>>left = 140245585033024
>>>status = 1
>>> #9  0x55e62c08f976 in slave_out (task=0x7f8d7846f740,
>>> f=0x7f8d7846bdd8, bb=0x7f8d7864ba90) at h2_task.c:177
>>>status = -512
>>>flush = 0
>>>blocking = 1
>>> #10 0x55e62c08fbfd in h2_filter_slave_output (filter=0x7f8d7846bdd8,
>>> brigade=0x7f8d7864ba90) at h2_task.c:370
>>>task = 0x7f8d7846f740
>>>status = 
>>> #11 0x55e62bffc01c in ap_content_length_filter (f=0x7f8d78472ca8,
>>> b=0x7f8d7864ba90) at protocol.c:1713
>>>r = 0x7f8d78471750
>>>ctx = 0x7f8d7864bc90
>>>e = 0x7f8d7864ba98
>>>eblock = (unknown: 2159446012)
>>> #12 0x55e62c04853d in deflate_out_filter (f=0x7f8d7864b650,
>>> bb=0x7f8d7864b8f8) at mod_deflate.c:961
>>>rv = -512
>>>r = 0x7f8d78471750
>>>ctx = 0x7f8d7864b9a8
>>>len = 8096
>>>blen = 1157663
>>>data = 0x7f8d7e40b000 "/*! jQuery v2.1.4 | (c) 2005, 2015 jQuery
>>> Foundation, Inc. | jquery.org/license
>>> */\n!function(a,b){\"object\"==typeof module&&\"object\"==typeof
>>> module.exports?module.exports=a.document?b(a,!0):function(a)"...
>>>c = 0x55e62c7a7498
>>> #13 0x55e62c044b1d in filter_harness (f=0x7f8d7846bc28, bb=0x80) at
>>> mod_filter.c:323
>>>ret = -512
>>>cachecontrol = 0xfe00 >> at address 0xfe00>
>>>filter = 0xe
>>> #14 0x55e62c002242 in default_handler (r=0x7f8d78471750) at core.c:4746
>>>c = 0x7f8d7846b720
>>>bb = 0x7f8d7864b8f8
>>>e = 0xfe00
>>>d = 0x7f8d7864dfa0
>>>fd = 0x7f8d7864b7b8
>>>bld_content_md5 = 2019866872
>>> #15 0x55e62c014130 in ap_run_handler 

Re: hanging apache httpd processes

[Stefan Eissing]

Here we go: https://github.com/icing/mod_h2/releases/tag/v1.10.2

> Am 18.04.2017 um 15:11 schrieb Stefan Eissing :
> 
> In transit. Just some minutes away...
> 
>> Am 18.04.2017 um 15:06 schrieb Stefan Priebe - Profihost AG 
>> :
>> 
>> 
>> Am 18.04.2017 um 15:03 schrieb Stefan Eissing:
>>> Stefan,
>>> 
>>> that is a 1.10.0, right? That was the first version without nested locking 
>>> and I fixed 2 possible dead locks in 1.10.1. 
>>> 
>>> I am about to release a 1.10.2 with added conformity checks and a fix for 
>>> client omitting EOF flags. Could you give that one a try?
>> 
>> Yes sure where is 1.10.2?
>> 
>> Stefan
>> 
>>> 
>>> -Stefan
>>> 
 Am 18.04.2017 um 14:57 schrieb Stefan Priebe - Profihost AG 
 :
 
 Hi,
 
 i saw that all of them are still serving one h2 connection.
 
 server-status:
 0-332375   42/64/181   G   30.09   1020776 214 1285.1  
 2.405.81
 h081217236127.dyn.cm.kabsi.at  h2  :443GET
 /wp-content/uploads/Bloglr21-8003asd.jpg HTTP/2.0
 
 And they all have Mode of operation => G
 
 thread apply all bt full shows:
 #0  0x7f8e80b687fc in __lll_lock_wait () from
 /lib/x86_64-linux-gnu/libpthread.so.0
 No symbol table info available.
 #1  0x7f8e80b6b6f8 in _L_cond_lock_886 () from
 /lib/x86_64-linux-gnu/libpthread.so.0
 No symbol table info available.
 #2  0x7f8e80b6b464 in __pthread_mutex_cond_lock () from
 /lib/x86_64-linux-gnu/libpthread.so.0
 No symbol table info available.
 #3  0x7f8e80b6650a in pthread_cond_timedwait@@GLIBC_2.3.2 () from
 /lib/x86_64-linux-gnu/libpthread.so.0
 No symbol table info available.
 #4  0x7f8e80d92322 in apr_thread_cond_timedwait
 (cond=0x7f8d7846bc70, mutex=0x7f8d7846bc20, timeout=6000) at
 locks/unix/thread_cond.c:89
  rv = 
  then = 
  abstime = {tv_sec = 1491843095, tv_nsec = 700198000}
 #5  0x55e62c0951c5 in r_wait_space (premain=,
 pbl=0x7f8de853da60, block=APR_BLOCK_READ, beam=0x7f8d7846bb08) at
 h2_bucket_beam.c:337
  status = 
 #6  append_bucket (pbl=0x7f8de853da60, block=APR_BLOCK_READ,
 b=0x7f8d7846dd38, beam=0x7f8d7846bb08) at h2_bucket_beam.c:776
  data = 0x7f8de853dac8 "\240\037"
  len = 0
  space_left = 0
  status = 
 #7  h2_beam_send (beam=0x7f8d7846bb08, sender_bb=0x7f8d7864ba90,
 block=APR_BLOCK_READ) at h2_bucket_beam.c:906
  force_report = 1
  b = 0x7f8d7846dd38
  status = 0
  bl = {mutex = 0x7f8d7846bc20, leave = 0x55e62c094250
 , leave_ctx = 0x0}
 #8  0x55e62c08f3aa in send_out (task=0x7f8d7846f740,
 bb=0x7f8d7864ba90, block=1) at h2_task.c:100
  written = 8096
  left = 140245585033024
  status = 1
 #9  0x55e62c08f976 in slave_out (task=0x7f8d7846f740,
 f=0x7f8d7846bdd8, bb=0x7f8d7864ba90) at h2_task.c:177
  status = -512
  flush = 0
  blocking = 1
 #10 0x55e62c08fbfd in h2_filter_slave_output (filter=0x7f8d7846bdd8,
 brigade=0x7f8d7864ba90) at h2_task.c:370
  task = 0x7f8d7846f740
  status = 
 #11 0x55e62bffc01c in ap_content_length_filter (f=0x7f8d78472ca8,
 b=0x7f8d7864ba90) at protocol.c:1713
  r = 0x7f8d78471750
  ctx = 0x7f8d7864bc90
  e = 0x7f8d7864ba98
  eblock = (unknown: 2159446012)
 #12 0x55e62c04853d in deflate_out_filter (f=0x7f8d7864b650,
 bb=0x7f8d7864b8f8) at mod_deflate.c:961
  rv = -512
  r = 0x7f8d78471750
  ctx = 0x7f8d7864b9a8
  len = 8096
  blen = 1157663
  data = 0x7f8d7e40b000 "/*! jQuery v2.1.4 | (c) 2005, 2015 jQuery
 Foundation, Inc. | jquery.org/license
 */\n!function(a,b){\"object\"==typeof module&&\"object\"==typeof
 module.exports?module.exports=a.document?b(a,!0):function(a)"...
  c = 0x55e62c7a7498
 #13 0x55e62c044b1d in filter_harness (f=0x7f8d7846bc28, bb=0x80) at
 mod_filter.c:323
  ret = -512
  cachecontrol = 0xfe00 >>> at address 0xfe00>
  filter = 0xe
 #14 0x55e62c002242 in default_handler (r=0x7f8d78471750) at core.c:4746
  c = 0x7f8d7846b720
  bb = 0x7f8d7864b8f8
  e = 0xfe00
  d = 0x7f8d7864dfa0
  fd = 0x7f8d7864b7b8
  bld_content_md5 = 2019866872
 #15 0x55e62c014130 in ap_run_handler (r=r@entry=0x7f8d78471750) at
 config.c:170
  pHook = 
  n = 11
  rv = -1
 #16 0x55e62c014679 in ap_invoke_handler (r=0x7f8d78471750) at
 config.c:434
  handler = 
  p = 
  result = 
  old_handler = 0x0
  ignore = 
 #17 0x55e62c04dcb2 in ap_process_async_request 

Re: hanging apache httpd processes

[Stefan Eissing]

In transit. Just some minutes away...

> Am 18.04.2017 um 15:06 schrieb Stefan Priebe - Profihost AG 
> :
> 
> 
> Am 18.04.2017 um 15:03 schrieb Stefan Eissing:
>> Stefan,
>> 
>> that is a 1.10.0, right? That was the first version without nested locking 
>> and I fixed 2 possible dead locks in 1.10.1. 
>> 
>> I am about to release a 1.10.2 with added conformity checks and a fix for 
>> client omitting EOF flags. Could you give that one a try?
> 
> Yes sure where is 1.10.2?
> 
> Stefan
> 
>> 
>> -Stefan
>> 
>>> Am 18.04.2017 um 14:57 schrieb Stefan Priebe - Profihost AG 
>>> :
>>> 
>>> Hi,
>>> 
>>> i saw that all of them are still serving one h2 connection.
>>> 
>>> server-status:
>>> 0-3 32375   42/64/181   G   30.09   1020776 214 1285.1  2.40
>>> 5.81
>>> h081217236127.dyn.cm.kabsi.at   h2  :443GET
>>> /wp-content/uploads/Bloglr21-8003asd.jpg HTTP/2.0
>>> 
>>> And they all have Mode of operation => G
>>> 
>>> thread apply all bt full shows:
>>> #0  0x7f8e80b687fc in __lll_lock_wait () from
>>> /lib/x86_64-linux-gnu/libpthread.so.0
>>> No symbol table info available.
>>> #1  0x7f8e80b6b6f8 in _L_cond_lock_886 () from
>>> /lib/x86_64-linux-gnu/libpthread.so.0
>>> No symbol table info available.
>>> #2  0x7f8e80b6b464 in __pthread_mutex_cond_lock () from
>>> /lib/x86_64-linux-gnu/libpthread.so.0
>>> No symbol table info available.
>>> #3  0x7f8e80b6650a in pthread_cond_timedwait@@GLIBC_2.3.2 () from
>>> /lib/x86_64-linux-gnu/libpthread.so.0
>>> No symbol table info available.
>>> #4  0x7f8e80d92322 in apr_thread_cond_timedwait
>>> (cond=0x7f8d7846bc70, mutex=0x7f8d7846bc20, timeout=6000) at
>>> locks/unix/thread_cond.c:89
>>>   rv = 
>>>   then = 
>>>   abstime = {tv_sec = 1491843095, tv_nsec = 700198000}
>>> #5  0x55e62c0951c5 in r_wait_space (premain=,
>>> pbl=0x7f8de853da60, block=APR_BLOCK_READ, beam=0x7f8d7846bb08) at
>>> h2_bucket_beam.c:337
>>>   status = 
>>> #6  append_bucket (pbl=0x7f8de853da60, block=APR_BLOCK_READ,
>>> b=0x7f8d7846dd38, beam=0x7f8d7846bb08) at h2_bucket_beam.c:776
>>>   data = 0x7f8de853dac8 "\240\037"
>>>   len = 0
>>>   space_left = 0
>>>   status = 
>>> #7  h2_beam_send (beam=0x7f8d7846bb08, sender_bb=0x7f8d7864ba90,
>>> block=APR_BLOCK_READ) at h2_bucket_beam.c:906
>>>   force_report = 1
>>>   b = 0x7f8d7846dd38
>>>   status = 0
>>>   bl = {mutex = 0x7f8d7846bc20, leave = 0x55e62c094250
>>> , leave_ctx = 0x0}
>>> #8  0x55e62c08f3aa in send_out (task=0x7f8d7846f740,
>>> bb=0x7f8d7864ba90, block=1) at h2_task.c:100
>>>   written = 8096
>>>   left = 140245585033024
>>>   status = 1
>>> #9  0x55e62c08f976 in slave_out (task=0x7f8d7846f740,
>>> f=0x7f8d7846bdd8, bb=0x7f8d7864ba90) at h2_task.c:177
>>>   status = -512
>>>   flush = 0
>>>   blocking = 1
>>> #10 0x55e62c08fbfd in h2_filter_slave_output (filter=0x7f8d7846bdd8,
>>> brigade=0x7f8d7864ba90) at h2_task.c:370
>>>   task = 0x7f8d7846f740
>>>   status = 
>>> #11 0x55e62bffc01c in ap_content_length_filter (f=0x7f8d78472ca8,
>>> b=0x7f8d7864ba90) at protocol.c:1713
>>>   r = 0x7f8d78471750
>>>   ctx = 0x7f8d7864bc90
>>>   e = 0x7f8d7864ba98
>>>   eblock = (unknown: 2159446012)
>>> #12 0x55e62c04853d in deflate_out_filter (f=0x7f8d7864b650,
>>> bb=0x7f8d7864b8f8) at mod_deflate.c:961
>>>   rv = -512
>>>   r = 0x7f8d78471750
>>>   ctx = 0x7f8d7864b9a8
>>>   len = 8096
>>>   blen = 1157663
>>>   data = 0x7f8d7e40b000 "/*! jQuery v2.1.4 | (c) 2005, 2015 jQuery
>>> Foundation, Inc. | jquery.org/license
>>> */\n!function(a,b){\"object\"==typeof module&&\"object\"==typeof
>>> module.exports?module.exports=a.document?b(a,!0):function(a)"...
>>>   c = 0x55e62c7a7498
>>> #13 0x55e62c044b1d in filter_harness (f=0x7f8d7846bc28, bb=0x80) at
>>> mod_filter.c:323
>>>   ret = -512
>>>   cachecontrol = 0xfe00 >> at address 0xfe00>
>>>   filter = 0xe
>>> #14 0x55e62c002242 in default_handler (r=0x7f8d78471750) at core.c:4746
>>>   c = 0x7f8d7846b720
>>>   bb = 0x7f8d7864b8f8
>>>   e = 0xfe00
>>>   d = 0x7f8d7864dfa0
>>>   fd = 0x7f8d7864b7b8
>>>   bld_content_md5 = 2019866872
>>> #15 0x55e62c014130 in ap_run_handler (r=r@entry=0x7f8d78471750) at
>>> config.c:170
>>>   pHook = 
>>>   n = 11
>>>   rv = -1
>>> #16 0x55e62c014679 in ap_invoke_handler (r=0x7f8d78471750) at
>>> config.c:434
>>>   handler = 
>>>   p = 
>>>   result = 
>>>   old_handler = 0x0
>>>   ignore = 
>>> #17 0x55e62c04dcb2 in ap_process_async_request (r=0x7f8d78471750) at
>>> http_request.c:436
>>>   c = 0x7f8d7846b720
>>>   access_status = 0
>>> #18 0x55e62c04de50 in ap_process_request (r=0x7f8d78471750) at
>>> http_request.c:471
>>>   bb = 0x7f8d7846bd80
>>>   c = 

Re: hanging apache httpd processes

[Stefan Priebe - Profihost AG]

Am 18.04.2017 um 15:03 schrieb Stefan Eissing:
> Stefan,
> 
> that is a 1.10.0, right? That was the first version without nested locking 
> and I fixed 2 possible dead locks in 1.10.1. 
> 
> I am about to release a 1.10.2 with added conformity checks and a fix for 
> client omitting EOF flags. Could you give that one a try?

Yes sure where is 1.10.2?

Stefan

> 
> -Stefan
> 
>> Am 18.04.2017 um 14:57 schrieb Stefan Priebe - Profihost AG 
>> :
>>
>> Hi,
>>
>> i saw that all of them are still serving one h2 connection.
>>
>> server-status:
>> 0-3  32375   42/64/181   G   30.09   1020776 214 1285.1  2.40
>> 5.81
>> h081217236127.dyn.cm.kabsi.ath2  :443GET
>> /wp-content/uploads/Bloglr21-8003asd.jpg HTTP/2.0
>>
>> And they all have Mode of operation => G
>>
>> thread apply all bt full shows:
>> #0  0x7f8e80b687fc in __lll_lock_wait () from
>> /lib/x86_64-linux-gnu/libpthread.so.0
>> No symbol table info available.
>> #1  0x7f8e80b6b6f8 in _L_cond_lock_886 () from
>> /lib/x86_64-linux-gnu/libpthread.so.0
>> No symbol table info available.
>> #2  0x7f8e80b6b464 in __pthread_mutex_cond_lock () from
>> /lib/x86_64-linux-gnu/libpthread.so.0
>> No symbol table info available.
>> #3  0x7f8e80b6650a in pthread_cond_timedwait@@GLIBC_2.3.2 () from
>> /lib/x86_64-linux-gnu/libpthread.so.0
>> No symbol table info available.
>> #4  0x7f8e80d92322 in apr_thread_cond_timedwait
>> (cond=0x7f8d7846bc70, mutex=0x7f8d7846bc20, timeout=6000) at
>> locks/unix/thread_cond.c:89
>>rv = 
>>then = 
>>abstime = {tv_sec = 1491843095, tv_nsec = 700198000}
>> #5  0x55e62c0951c5 in r_wait_space (premain=,
>> pbl=0x7f8de853da60, block=APR_BLOCK_READ, beam=0x7f8d7846bb08) at
>> h2_bucket_beam.c:337
>>status = 
>> #6  append_bucket (pbl=0x7f8de853da60, block=APR_BLOCK_READ,
>> b=0x7f8d7846dd38, beam=0x7f8d7846bb08) at h2_bucket_beam.c:776
>>data = 0x7f8de853dac8 "\240\037"
>>len = 0
>>space_left = 0
>>status = 
>> #7  h2_beam_send (beam=0x7f8d7846bb08, sender_bb=0x7f8d7864ba90,
>> block=APR_BLOCK_READ) at h2_bucket_beam.c:906
>>force_report = 1
>>b = 0x7f8d7846dd38
>>status = 0
>>bl = {mutex = 0x7f8d7846bc20, leave = 0x55e62c094250
>> , leave_ctx = 0x0}
>> #8  0x55e62c08f3aa in send_out (task=0x7f8d7846f740,
>> bb=0x7f8d7864ba90, block=1) at h2_task.c:100
>>written = 8096
>>left = 140245585033024
>>status = 1
>> #9  0x55e62c08f976 in slave_out (task=0x7f8d7846f740,
>> f=0x7f8d7846bdd8, bb=0x7f8d7864ba90) at h2_task.c:177
>>status = -512
>>flush = 0
>>blocking = 1
>> #10 0x55e62c08fbfd in h2_filter_slave_output (filter=0x7f8d7846bdd8,
>> brigade=0x7f8d7864ba90) at h2_task.c:370
>>task = 0x7f8d7846f740
>>status = 
>> #11 0x55e62bffc01c in ap_content_length_filter (f=0x7f8d78472ca8,
>> b=0x7f8d7864ba90) at protocol.c:1713
>>r = 0x7f8d78471750
>>ctx = 0x7f8d7864bc90
>>e = 0x7f8d7864ba98
>>eblock = (unknown: 2159446012)
>> #12 0x55e62c04853d in deflate_out_filter (f=0x7f8d7864b650,
>> bb=0x7f8d7864b8f8) at mod_deflate.c:961
>>rv = -512
>>r = 0x7f8d78471750
>>ctx = 0x7f8d7864b9a8
>>len = 8096
>>blen = 1157663
>>data = 0x7f8d7e40b000 "/*! jQuery v2.1.4 | (c) 2005, 2015 jQuery
>> Foundation, Inc. | jquery.org/license
>> */\n!function(a,b){\"object\"==typeof module&&\"object\"==typeof
>> module.exports?module.exports=a.document?b(a,!0):function(a)"...
>>c = 0x55e62c7a7498
>> #13 0x55e62c044b1d in filter_harness (f=0x7f8d7846bc28, bb=0x80) at
>> mod_filter.c:323
>>ret = -512
>>cachecontrol = 0xfe00 > at address 0xfe00>
>>filter = 0xe
>> #14 0x55e62c002242 in default_handler (r=0x7f8d78471750) at core.c:4746
>>c = 0x7f8d7846b720
>>bb = 0x7f8d7864b8f8
>>e = 0xfe00
>>d = 0x7f8d7864dfa0
>>fd = 0x7f8d7864b7b8
>>bld_content_md5 = 2019866872
>> #15 0x55e62c014130 in ap_run_handler (r=r@entry=0x7f8d78471750) at
>> config.c:170
>>pHook = 
>>n = 11
>>rv = -1
>> #16 0x55e62c014679 in ap_invoke_handler (r=0x7f8d78471750) at
>> config.c:434
>>handler = 
>>p = 
>>result = 
>>old_handler = 0x0
>>ignore = 
>> #17 0x55e62c04dcb2 in ap_process_async_request (r=0x7f8d78471750) at
>> http_request.c:436
>>c = 0x7f8d7846b720
>>access_status = 0
>> #18 0x55e62c04de50 in ap_process_request (r=0x7f8d78471750) at
>> http_request.c:471
>>bb = 0x7f8d7846bd80
>>c = 0x7f8d7846b720
>>rv = -512
>> #19 0x55e62c08ec32 in h2_task_process_request (c=,
>> task=) at h2_task.c:666
>>req = 0x7f8d78471750
>>cs = 0x7f8d7846bd80
>>r = 0x7f8d78471750
>> #20 

Re: hanging apache httpd processes

[Stefan Eissing]

Stefan,

that is a 1.10.0, right? That was the first version without nested locking and 
I fixed 2 possible dead locks in 1.10.1. 

I am about to release a 1.10.2 with added conformity checks and a fix for 
client omitting EOF flags. Could you give that one a try?

-Stefan

> Am 18.04.2017 um 14:57 schrieb Stefan Priebe - Profihost AG 
> :
> 
> Hi,
> 
> i saw that all of them are still serving one h2 connection.
> 
> server-status:
> 0-3   32375   42/64/181   G   30.09   1020776 214 1285.1  2.40
> 5.81
> h081217236127.dyn.cm.kabsi.at h2  :443GET
> /wp-content/uploads/Bloglr21-8003asd.jpg HTTP/2.0
> 
> And they all have Mode of operation => G
> 
> thread apply all bt full shows:
> #0  0x7f8e80b687fc in __lll_lock_wait () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> No symbol table info available.
> #1  0x7f8e80b6b6f8 in _L_cond_lock_886 () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> No symbol table info available.
> #2  0x7f8e80b6b464 in __pthread_mutex_cond_lock () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> No symbol table info available.
> #3  0x7f8e80b6650a in pthread_cond_timedwait@@GLIBC_2.3.2 () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> No symbol table info available.
> #4  0x7f8e80d92322 in apr_thread_cond_timedwait
> (cond=0x7f8d7846bc70, mutex=0x7f8d7846bc20, timeout=6000) at
> locks/unix/thread_cond.c:89
>rv = 
>then = 
>abstime = {tv_sec = 1491843095, tv_nsec = 700198000}
> #5  0x55e62c0951c5 in r_wait_space (premain=,
> pbl=0x7f8de853da60, block=APR_BLOCK_READ, beam=0x7f8d7846bb08) at
> h2_bucket_beam.c:337
>status = 
> #6  append_bucket (pbl=0x7f8de853da60, block=APR_BLOCK_READ,
> b=0x7f8d7846dd38, beam=0x7f8d7846bb08) at h2_bucket_beam.c:776
>data = 0x7f8de853dac8 "\240\037"
>len = 0
>space_left = 0
>status = 
> #7  h2_beam_send (beam=0x7f8d7846bb08, sender_bb=0x7f8d7864ba90,
> block=APR_BLOCK_READ) at h2_bucket_beam.c:906
>force_report = 1
>b = 0x7f8d7846dd38
>status = 0
>bl = {mutex = 0x7f8d7846bc20, leave = 0x55e62c094250
> , leave_ctx = 0x0}
> #8  0x55e62c08f3aa in send_out (task=0x7f8d7846f740,
> bb=0x7f8d7864ba90, block=1) at h2_task.c:100
>written = 8096
>left = 140245585033024
>status = 1
> #9  0x55e62c08f976 in slave_out (task=0x7f8d7846f740,
> f=0x7f8d7846bdd8, bb=0x7f8d7864ba90) at h2_task.c:177
>status = -512
>flush = 0
>blocking = 1
> #10 0x55e62c08fbfd in h2_filter_slave_output (filter=0x7f8d7846bdd8,
> brigade=0x7f8d7864ba90) at h2_task.c:370
>task = 0x7f8d7846f740
>status = 
> #11 0x55e62bffc01c in ap_content_length_filter (f=0x7f8d78472ca8,
> b=0x7f8d7864ba90) at protocol.c:1713
>r = 0x7f8d78471750
>ctx = 0x7f8d7864bc90
>e = 0x7f8d7864ba98
>eblock = (unknown: 2159446012)
> #12 0x55e62c04853d in deflate_out_filter (f=0x7f8d7864b650,
> bb=0x7f8d7864b8f8) at mod_deflate.c:961
>rv = -512
>r = 0x7f8d78471750
>ctx = 0x7f8d7864b9a8
>len = 8096
>blen = 1157663
>data = 0x7f8d7e40b000 "/*! jQuery v2.1.4 | (c) 2005, 2015 jQuery
> Foundation, Inc. | jquery.org/license
> */\n!function(a,b){\"object\"==typeof module&&\"object\"==typeof
> module.exports?module.exports=a.document?b(a,!0):function(a)"...
>c = 0x55e62c7a7498
> #13 0x55e62c044b1d in filter_harness (f=0x7f8d7846bc28, bb=0x80) at
> mod_filter.c:323
>ret = -512
>cachecontrol = 0xfe00  at address 0xfe00>
>filter = 0xe
> #14 0x55e62c002242 in default_handler (r=0x7f8d78471750) at core.c:4746
>c = 0x7f8d7846b720
>bb = 0x7f8d7864b8f8
>e = 0xfe00
>d = 0x7f8d7864dfa0
>fd = 0x7f8d7864b7b8
>bld_content_md5 = 2019866872
> #15 0x55e62c014130 in ap_run_handler (r=r@entry=0x7f8d78471750) at
> config.c:170
>pHook = 
>n = 11
>rv = -1
> #16 0x55e62c014679 in ap_invoke_handler (r=0x7f8d78471750) at
> config.c:434
>handler = 
>p = 
>result = 
>old_handler = 0x0
>ignore = 
> #17 0x55e62c04dcb2 in ap_process_async_request (r=0x7f8d78471750) at
> http_request.c:436
>c = 0x7f8d7846b720
>access_status = 0
> #18 0x55e62c04de50 in ap_process_request (r=0x7f8d78471750) at
> http_request.c:471
>bb = 0x7f8d7846bd80
>c = 0x7f8d7846b720
>rv = -512
> #19 0x55e62c08ec32 in h2_task_process_request (c=,
> task=) at h2_task.c:666
>req = 0x7f8d78471750
>cs = 0x7f8d7846bd80
>r = 0x7f8d78471750
> #20 h2_task_process_conn (c=0x7f8d7846bc28) at h2_task.c:713
>ctx = 0x7f8d7846bd80
> #21 0x55e62c01e130 in ap_run_process_connection (c=0x7f8d7846b720)
> at connection.c:42
>pHook = 
>n = 0
>rv = -1
> #22 

Re: hanging apache httpd processes

[Stefan Priebe - Profihost AG]

Hi,

i saw that all of them are still serving one h2 connection.

server-status:
0-3 32375   42/64/181   G   30.09   1020776 214 1285.1  2.40
5.81
h081217236127.dyn.cm.kabsi.at   h2  :443GET
/wp-content/uploads/Bloglr21-8003asd.jpg HTTP/2.0

And they all have Mode of operation => G

thread apply all bt full shows:
#0  0x7f8e80b687fc in __lll_lock_wait () from
/lib/x86_64-linux-gnu/libpthread.so.0
No symbol table info available.
#1  0x7f8e80b6b6f8 in _L_cond_lock_886 () from
/lib/x86_64-linux-gnu/libpthread.so.0
No symbol table info available.
#2  0x7f8e80b6b464 in __pthread_mutex_cond_lock () from
/lib/x86_64-linux-gnu/libpthread.so.0
No symbol table info available.
#3  0x7f8e80b6650a in pthread_cond_timedwait@@GLIBC_2.3.2 () from
/lib/x86_64-linux-gnu/libpthread.so.0
No symbol table info available.
#4  0x7f8e80d92322 in apr_thread_cond_timedwait
(cond=0x7f8d7846bc70, mutex=0x7f8d7846bc20, timeout=6000) at
locks/unix/thread_cond.c:89
rv = 
then = 
abstime = {tv_sec = 1491843095, tv_nsec = 700198000}
#5  0x55e62c0951c5 in r_wait_space (premain=,
pbl=0x7f8de853da60, block=APR_BLOCK_READ, beam=0x7f8d7846bb08) at
h2_bucket_beam.c:337
status = 
#6  append_bucket (pbl=0x7f8de853da60, block=APR_BLOCK_READ,
b=0x7f8d7846dd38, beam=0x7f8d7846bb08) at h2_bucket_beam.c:776
data = 0x7f8de853dac8 "\240\037"
len = 0
space_left = 0
status = 
#7  h2_beam_send (beam=0x7f8d7846bb08, sender_bb=0x7f8d7864ba90,
block=APR_BLOCK_READ) at h2_bucket_beam.c:906
force_report = 1
b = 0x7f8d7846dd38
status = 0
bl = {mutex = 0x7f8d7846bc20, leave = 0x55e62c094250
, leave_ctx = 0x0}
#8  0x55e62c08f3aa in send_out (task=0x7f8d7846f740,
bb=0x7f8d7864ba90, block=1) at h2_task.c:100
written = 8096
left = 140245585033024
status = 1
#9  0x55e62c08f976 in slave_out (task=0x7f8d7846f740,
f=0x7f8d7846bdd8, bb=0x7f8d7864ba90) at h2_task.c:177
status = -512
flush = 0
blocking = 1
#10 0x55e62c08fbfd in h2_filter_slave_output (filter=0x7f8d7846bdd8,
brigade=0x7f8d7864ba90) at h2_task.c:370
task = 0x7f8d7846f740
status = 
#11 0x55e62bffc01c in ap_content_length_filter (f=0x7f8d78472ca8,
b=0x7f8d7864ba90) at protocol.c:1713
r = 0x7f8d78471750
ctx = 0x7f8d7864bc90
e = 0x7f8d7864ba98
eblock = (unknown: 2159446012)
#12 0x55e62c04853d in deflate_out_filter (f=0x7f8d7864b650,
bb=0x7f8d7864b8f8) at mod_deflate.c:961
rv = -512
r = 0x7f8d78471750
ctx = 0x7f8d7864b9a8
len = 8096
blen = 1157663
data = 0x7f8d7e40b000 "/*! jQuery v2.1.4 | (c) 2005, 2015 jQuery
Foundation, Inc. | jquery.org/license
*/\n!function(a,b){\"object\"==typeof module&&\"object\"==typeof
module.exports?module.exports=a.document?b(a,!0):function(a)"...
c = 0x55e62c7a7498
#13 0x55e62c044b1d in filter_harness (f=0x7f8d7846bc28, bb=0x80) at
mod_filter.c:323
ret = -512
cachecontrol = 0xfe00 
filter = 0xe
#14 0x55e62c002242 in default_handler (r=0x7f8d78471750) at core.c:4746
c = 0x7f8d7846b720
bb = 0x7f8d7864b8f8
e = 0xfe00
d = 0x7f8d7864dfa0
fd = 0x7f8d7864b7b8
bld_content_md5 = 2019866872
#15 0x55e62c014130 in ap_run_handler (r=r@entry=0x7f8d78471750) at
config.c:170
pHook = 
n = 11
rv = -1
#16 0x55e62c014679 in ap_invoke_handler (r=0x7f8d78471750) at
config.c:434
handler = 
p = 
result = 
old_handler = 0x0
ignore = 
#17 0x55e62c04dcb2 in ap_process_async_request (r=0x7f8d78471750) at
http_request.c:436
c = 0x7f8d7846b720
access_status = 0
#18 0x55e62c04de50 in ap_process_request (r=0x7f8d78471750) at
http_request.c:471
bb = 0x7f8d7846bd80
c = 0x7f8d7846b720
rv = -512
#19 0x55e62c08ec32 in h2_task_process_request (c=,
task=) at h2_task.c:666
req = 0x7f8d78471750
cs = 0x7f8d7846bd80
r = 0x7f8d78471750
#20 h2_task_process_conn (c=0x7f8d7846bc28) at h2_task.c:713
ctx = 0x7f8d7846bd80
#21 0x55e62c01e130 in ap_run_process_connection (c=0x7f8d7846b720)
at connection.c:42
pHook = 
n = 0
rv = -1
#22 0x55e62c09012b in h2_task_do (task=0x7f8d7846f740,
thread=0x55e62cb0ce70, worker_id=0) at h2_task.c:623
c = 0x7f8d7846b720
#23 0x55e62c093619 in slot_run (thread=0x55e62cb0ce70,
wctx=0x55e62c8b9268) at h2_workers.c:217
slot = 0x55e62c8b9268
#24 0x7f8e80b620a4 in start_thread () from
/lib/x86_64-linux-gnu/libpthread.so.0
No symbol table info available.
#25 0x7f8e8089762d in clone () from /lib/x86_64-linux-gnu/libc.so.6
No symbol table info available.

Thread 3 (Thread 0x7f8da678c700 (LWP 28251)):
#0  0x7f8e80b687fc in __lll_lock_wait () from

Re: hanging apache httpd processes

[Stefan Eissing]

What Eric said.

With the changes in http2 worker scheduling, if I introduced a bug
in child exit there, I'd expect it to trigger via workers_pool_cleanup()
in h2_workers.c

-Stefan

> Am 18.04.2017 um 14:47 schrieb Eric Covener :
> 
> On Tue, Apr 18, 2017 at 8:43 AM, Stefan Priebe - Profihost AG
>  wrote:
>> bt of such a process shows:
>> (gdb) bt
>> #0  0x7f5df74f64db in pthread_join () from
>> /lib/x86_64-linux-gnu/libpthread.so.0
> 
> Need to see the other threads in the process, this one is just waiting
> for the others to complete.
> 
> "thread apply all bt full"
> 
> 
> -- 
> Eric Covener
> cove...@gmail.com

Re: hanging apache httpd processes

[Eric Covener]

On Tue, Apr 18, 2017 at 8:43 AM, Stefan Priebe - Profihost AG
 wrote:
> bt of such a process shows:
> (gdb) bt
> #0  0x7f5df74f64db in pthread_join () from
> /lib/x86_64-linux-gnu/libpthread.so.0

Need to see the other threads in the process, this one is just waiting
for the others to complete.

 "thread apply all bt full"


-- 
Eric Covener
cove...@gmail.com

hanging apache httpd processes

[Stefan Priebe - Profihost AG]

Hi,

not sure whether this is related to mod_http2 v1.10.0 or is something else.

I've seen two servers where old httpd processes get stuck.

server-status looks like this:
SlotPID StoppingConnections Threads Async connections
total   accepting   busyidlewriting keep-alive  closing
0   27399   yes (old gen)   3   no  0   0   0   0   0
1   879 yes (old gen)   1   no  0   0   0   0   0
2   11609   yes (old gen)   2   no  0   0   0   0   0
3   9142yes (old gen)   2   no  0   0   0   0   0
4   11098   yes (old gen)   3   no  0   0   0   0   0
5   27400   yes (old gen)   1   no  0   0   0   0   0
6   16445   yes (old gen)   1   no  0   0   0   0   0
7   16446   yes (old gen)   1   no  0   0   0   0   0
8   7388yes (old gen)   4   no  0   0   0   0   0
9   24072   yes (old gen)   2   no  0   0   0   0   0
10  28673   yes (old gen)   2   no  0   0   0   0   0
11  7389yes (old gen)   3   no  0   0   0   0   0
12  11075   yes (old gen)   8   no  0   0   0   0   0
13  11076   yes (old gen)   6   no  0   0   0   0   0
14  26613   yes (old gen)   5   no  0   0   0   0   0
15  737 no  1   yes 8   192 0   0   1
16  739 no  3   yes 13  187 0   0   1
17  24059   yes (old gen)   1   no  0   0   0   0   0
18  26002   yes (old gen)   7   no  0   0   0   0   0
19  26003   yes (old gen)   4   no  0   0   0   0   0
20  2473yes (old gen)   2   no  0   0   0   0   0
21  3010yes (old gen)   4   no  0   0   0   0   0
22  8847yes (old gen)   3   no  0   0   0   0   0
Sum 23  21  69  21  379 0   0   2

bt of such a process shows:
(gdb) bt
#0  0x7f5df74f64db in pthread_join () from
/lib/x86_64-linux-gnu/libpthread.so.0
#1  0x7f5df77314ab in apr_thread_join (retval=0x7ffc0878f424,
thd=0x7f5d580dc7c8) at threadproc/unix/thread.c:217
#2  0x555c1dc9deb3 in join_workers (listener=0x7f5d580dc7c8,
threads=0x555c1f3d4310) at event.c:2376
#3  0x555c1dc9e25a in child_main (child_num_arg=0,
child_bucket=524108560) at event.c:2574
#4  0x555c1dd68fe0 in make_child (s=0x7f5cf673c9d0, slot=0,
bucket=0) at event.c:2646
#5  0x555c1dd6950c in server_main_loop (num_buckets=,
remaining_children_to_start=) at event.c:2916
#6  event_run (_pconf=0x7f5cf673c9d0, plog=0x0, s=0xc8) at event.c:3035
#7  0x555c1dca642e in ap_run_mpm (pconf=0x555c1f031138,
plog=0x555c1f072bb8, s=0x555c1f0625a8) at mpm_common.c:94
#8  0x555c1dc9ed15 in main (argc=2, argv=0x7ffc0878f828) at main.c:783


An strace is like this one:
strace -ff -p 32375
Process 32375 attached with 303 threads
[pid   856] futex(0x7f5ce8003c60, FUTEX_WAIT_PRIVATE, 2, NULL

[pid   486] futex(0x555c1f3b17d4, FUTEX_WAIT_PRIVATE, 1, NULL

[pid   857] epoll_wait(14,  
[pid   484] futex(0x555c1f3b174c, FUTEX_WAIT_PRIVATE, 1, NULL

[pid   482] futex(0x555c1f3b16c4, FUTEX_WAIT_PRIVATE, 1, NULL

[pid   481] futex(0x555c1f3b163c, FUTEX_WAIT_PRIVATE, 1, NULL

...


Greets,
Stefan