Re: [Bug 60956] Event MPM listener thread may get blocked by SSL shutdowns

Thanks for testing and verifying the fix, Stefan! > Am 31.07.2017 um 11:32 schrieb Stefan Priebe - Profihost AG > : > > 4tr i was able to fix this by mod_h2 v1.10.10 > > Greets, > Stefan > > Am 25.07.2017 um 15:40 schrieb Stefan Eissing: >> Well, if the customer could

Re: [Bug 60956] Event MPM listener thread may get blocked by SSL shutdowns

4tr i was able to fix this by mod_h2 v1.10.10 Greets, Stefan Am 25.07.2017 um 15:40 schrieb Stefan Eissing: > Well, if the customer could reproduce this at a > > LogLevel http2:trace2 > > that would help. > >> Am 25.07.2017 um 15:38 schrieb Stefan Priebe - Profihost AG >>

Re: [Bug 60956] Event MPM listener thread may get blocked by SSL shutdowns

I am waiting to hear back from the peeps that opened the github issue. From how I read their logs, the patch should help them. Will report what they say. -Stefan > Am 25.07.2017 um 15:40 schrieb Stefan Eissing : > > Well, if the customer could reproduce this at a

Re: [Bug 60956] Event MPM listener thread may get blocked by SSL shutdowns

Well, if the customer could reproduce this at a LogLevel http2:trace2 that would help. > Am 25.07.2017 um 15:38 schrieb Stefan Priebe - Profihost AG > : > > Hello Stefan, > > thanks for the patch. No it does not solve the problem our customer is > seeing. > > What

Re: [Bug 60956] Event MPM listener thread may get blocked by SSL shutdowns

Hello Stefan, thanks for the patch. No it does not solve the problem our customer is seeing. What kind of details / logs you need? Greets, Stefan Am 25.07.2017 um 11:59 schrieb Stefan Eissing: > The issue was opened here: https://github.com/icing/mod_h2/issues/143 > > I made a patch that i

Re: [Bug 60956] Event MPM listener thread may get blocked by SSL shutdowns

The issue was opened here: https://github.com/icing/mod_h2/issues/143 I made a patch that i hope addresses the problem. The 2.4.x version I attach to this mail. Thanks! Stefan h2_stream_stall_2.4.x-v0.diff Description: Binary data > Am 25.07.2017 um 08:13 schrieb Stefan Priebe - Profihost

Re: [Bug 60956] Event MPM listener thread may get blocked by SSL shutdowns

Am 24.07.2017 um 23:06 schrieb Stefan Eissing: > I have another report of request getting stuck, resulting in the error you > noticed. Will look tomorrow and report back here what I find. Thanks, if you need any logs. Pleae ask. Stefan > >> Am 24.07.2017 um 22:20 schrieb Stefan Priebe -

Re: [Bug 60956] Event MPM listener thread may get blocked by SSL shutdowns

I have another report of request getting stuck, resulting in the error you noticed. Will look tomorrow and report back here what I find. > Am 24.07.2017 um 22:20 schrieb Stefan Priebe - Profihost AG > : > > Hello all, > > currently 8 hours of testing without any issues.

Re: [Bug 60956] Event MPM listener thread may get blocked by SSL shutdowns

Hello all, currently 8 hours of testing without any issues. @Stefan i've most probably another issue with http2 where some elements of the page are sometimes missing and the connection results in ERR_CONNECTION_CLOSED after 60s. What kind of details do you need? Greets, Stefan Am 22.07.2017 um

Re: [Bug 60956] Event MPM listener thread may get blocked by SSL shutdowns

First test with version five looks good so far will continue extensive testing tomorrow. Greets, Stefan Excuse my typo sent from my mobile phone. > Am 22.07.2017 um 13:35 schrieb Yann Ylavic : > >> On Sat, Jul 22, 2017 at 2:18 AM, Yann Ylavic

Re: [Bug 60956] Event MPM listener thread may get blocked by SSL shutdowns

On Sat, Jul 22, 2017 at 2:18 AM, Yann Ylavic wrote: > On Fri, Jul 21, 2017 at 10:31 PM, Stefan Priebe - Profihost AG > wrote: >> >> your new defer linger V3 deadlocked as well. >> >> GDB traces: >> https://www.apaste.info/LMfJ > > This shows the

Re: [Bug 60956] Event MPM listener thread may get blocked by SSL shutdowns

And to answer myself: no, the v3 patch does not expose anything when running in h2fuzz. > Am 22.07.2017 um 07:17 schrieb Stefan Eissing : > > Profihost, where bugs come to die! > > I am currently fully overloaded, but it would be interesting to check how the >

Re: [Bug 60956] Event MPM listener thread may get blocked by SSL shutdowns

Profihost, where bugs come to die! I am currently fully overloaded, but it would be interesting to check how the previous versions of the patch fare in a h2fuzz setup. -Stefan > Am 22.07.2017 um 02:18 schrieb Yann Ylavic : > > On Fri, Jul 21, 2017 at 10:31 PM, Stefan

Re: [Bug 60956] Event MPM listener thread may get blocked by SSL shutdowns

On Fri, Jul 21, 2017 at 10:31 PM, Stefan Priebe - Profihost AG wrote: > > your new defer linger V3 deadlocked as well. > > GDB traces: > https://www.apaste.info/LMfJ This shows the listener thread waiting for a worker while there are many available. My mistake, the worker

Re: [Bug 60956] Event MPM listener thread may get blocked by SSL shutdowns

Hello Yann, your new defer linger V3 deadlocked as well. GDB traces: https://www.apaste.info/LMfJ But this time i have no fullstatus for you as the apache didn't serve any connections at all anymore. But even before i did NOT see those strange values for closing connections. Thanks! Greets,

Re: [Bug 60956] Event MPM listener thread may get blocked by SSL shutdowns

Hello Yann, i downloaded V3. Can't guarantee when i can test. May be today or on monday. Greets, Stefan Am 21.07.2017 um 01:08 schrieb Yann Ylavic: > On Thu, Jul 20, 2017 at 12:48 PM, Stefan Priebe - Profihost AG > wrote: >> V3 didn't help. > > I just posted a new patch

Re: [Bug 60956] Event MPM listener thread may get blocked by SSL shutdowns

> So, should we favor the draining of defer_linger_chain as much workers > as necessary like the current patch, or should we have as few workers > as possible and not start new workers in loops with no effect on > defer_linger_chain? I think the fewer workers option could lead to hard to debug

Re: [Bug 60956] Event MPM listener thread may get blocked by SSL shutdowns

> Also, it seems that in the deferred lingering case we should probaly > shorten the socket timeout before calling (and possibly blocking on) > ap_start_lingering_close()'s hooks/flush, since we likely come from a > time-up already... +1

Re: [Bug 60956] Event MPM listener thread may get blocked by SSL shutdowns

On Fri, Jul 21, 2017 at 3:07 PM, Luca Toscano wrote: >> >> To prevent starvation of deferred lingering closes, the listener may >> create a worker at the of its loop, when/if the chain is (fully) >> filled. > > IIUC the trick is to run "(have_idle_worker &&

Re: [Bug 60956] Event MPM listener thread may get blocked by SSL shutdowns

Hi Yann, 2017-07-21 1:05 GMT+02:00 Yann Ylavic : > On Fri, Jul 14, 2017 at 9:52 PM, Yann Ylavic wrote: > > > > So overall, this patch may introduce the need for more workers than > > before, what was (wrongly) done by the listener thread has to be

Re: [Bug 60956] Event MPM listener thread may get blocked by SSL shutdowns

2017-07-21 1:16 GMT+02:00 Yann Ylavic : > On Fri, Jul 21, 2017 at 1:05 AM, Postmaster > wrote: > > This message was created automatically by mail delivery software. Your > email message was not delivered as is to the intended recipients

Re: [Bug 60956] Event MPM listener thread may get blocked by SSL shutdowns

On Fri, Jul 21, 2017 at 1:08 AM, Yann Ylavic wrote: > On Thu, Jul 20, 2017 at 12:48 PM, Stefan Priebe - Profihost AG > wrote: >> V3 didn't help. > > I just posted a new patch in this thread, with a new approach which I > think is better anyway. > >

Re: [Bug 60956] Event MPM listener thread may get blocked by SSL shutdowns

On Fri, Jul 21, 2017 at 1:05 AM, Postmaster wrote: > This message was created automatically by mail delivery software. Your email > message was not delivered as is to the intended recipients because malware > was detected in one or more attachments included

Re: [Bug 60956] Event MPM listener thread may get blocked by SSL shutdowns

On Thu, Jul 20, 2017 at 12:48 PM, Stefan Priebe - Profihost AG wrote: > V3 didn't help. I just posted a new patch in this thread, with a new approach which I think is better anyway. Would you mind testing it in your environment? Regards, Yann.

Re: [Bug 60956] Event MPM listener thread may get blocked by SSL shutdowns

On Fri, Jul 14, 2017 at 9:52 PM, Yann Ylavic wrote: > > So overall, this patch may introduce the need for more workers than > before, what was (wrongly) done by the listener thread has to be done > somewhere anyway... That patch didn't work (as reported by Stefan Pribe) and

Re: [Bug 60956] Event MPM listener thread may get blocked by SSL shutdowns

On Fri, Jul 14, 2017 at 9:52 PM, Yann Ylavic wrote: > > So overall, this patch may introduce the need for more workers than > before, what was (wrongly) done by the listener thread has to be done > somewhere anyway... That patch didn't work (as reported by Stefan Pribe) and

Re: [Bug 60956] Event MPM listener thread may get blocked by SSL shutdowns

Yes: Slot PID Stopping ConnectionsThreads Async connections total accepting busy idle writing keep-alive closing 03614 no 1 no4196 0 0 4294966701 13615 no 0 no5195 0 0 4294966697 2

Re: [Bug 60956] Event MPM listener thread may get blocked by SSL shutdowns

On Thu, Jul 20, 2017 at 2:58 PM, Stefan Priebe - Profihost AG wrote: > Yes it looks the same but I can't tell if it is. > > Here's a backtrace from V3: > https://apaste.info/Aw0r Thanks Stefan, how about mod_status, still some strange entries? Regards, Yann.

Re: [Bug 60956] Event MPM listener thread may get blocked by SSL shutdowns

Yes it looks the same but I can't tell if it is. Here's a backtrace from V3: https://apaste.info/Aw0r Greets, Stefan Excuse my typo sent from my mobile phone. > Am 20.07.2017 um 13:01 schrieb Yann Ylavic : > > On Thu, Jul 20, 2017 at 12:48 PM, Stefan Priebe - Profihost

Re: [Bug 60956] Event MPM listener thread may get blocked by SSL shutdowns

On Thu, Jul 20, 2017 at 12:48 PM, Stefan Priebe - Profihost AG wrote: > V3 didn't help. Will post a new gdb backtrace soon takes some time as I'm on > holiday. Thanks Stefan, still some/the same issue in status? Regards, Yann.

Re: [Bug 60956] Event MPM listener thread may get blocked by SSL shutdowns

V3 didn't help. Will post a new gdb backtrace soon takes some time as I'm on holiday. Stefan Excuse my typo sent from my mobile phone. > Am 20.07.2017 um 01:26 schrieb Yann Ylavic : > > On Wed, Jul 19, 2017 at 11:14 PM, Stefan Priebe - Profihost AG >

Re: [Bug 60956] Event MPM listener thread may get blocked by SSL shutdowns

Am 20.07.2017 um 01:26 schrieb Yann Ylavic: > On Wed, Jul 19, 2017 at 11:14 PM, Stefan Priebe - Profihost AG > wrote: >> Am 19.07.2017 um 22:46 schrieb Yann Ylavic: >>> >>> Attached is a v2 if you feel confident enough, still ;) >> >> Thanks, yes i will. > > If you

Re: [Bug 60956] Event MPM listener thread may get blocked by SSL shutdowns

On Wed, Jul 19, 2017 at 11:14 PM, Stefan Priebe - Profihost AG wrote: > Am 19.07.2017 um 22:46 schrieb Yann Ylavic: >> >> Attached is a v2 if you feel confident enough, still ;) > > Thanks, yes i will. If you managed to install v2 already you may want to ignore this new

Re: [Bug 60956] Event MPM listener thread may get blocked by SSL shutdowns

Am 19.07.2017 um 22:46 schrieb Yann Ylavic: > Hi Stefan, > > thanks for testing again! > > On Wed, Jul 19, 2017 at 7:42 PM, Stefan Priebe - Profihost AG > wrote: >> >> What looks strange >> from a first view is that async connections closing has very high and >> strange

Re: [Bug 60956] Event MPM listener thread may get blocked by SSL shutdowns

Hi Stefan, thanks for testing again! On Wed, Jul 19, 2017 at 7:42 PM, Stefan Priebe - Profihost AG wrote: > > What looks strange > from a first view is that async connections closing has very high and > strange values: > 4294967211 Indeed, I messed up with mpm_event's

Re: [Bug 60956] Event MPM listener thread may get blocked by SSL shutdowns

On Wed, Jul 19, 2017 at 2:25 PM, Stefan Priebe - Profihost AG wrote: > Hello, > > here we go: > > This one is from a server where the first httpd process got stuck: > >Slot PID Stopping ConnectionsThreads Async connections >total

Re: [Bug 60956] Event MPM listener thread may get blocked by SSL shutdowns

Hello, here we go: This one is from a server where the first httpd process got stuck: Slot PID Stopping ConnectionsThreads Async connections total accepting busy idle writing keep-alive closing 031675 no 0 no0200 0 0

Re: [Bug 60956] Event MPM listener thread may get blocked by SSL shutdowns

Hello Luca, i need to wait until a machine is crashing again. What looks strange from a first view is that async connections closing has very high and strange values: 4294967211 Even a not yet crashed system has those: Slot PID Stopping ConnectionsThreads Async connections

Re: [Bug 60956] Event MPM listener thread may get blocked by SSL shutdowns

Hello Stefan, 2017-07-19 17:05 GMT+02:00 Stefan Priebe - Profihost AG < s.pri...@profihost.ag>: > > Am 19.07.2017 um 16:59 schrieb Stefan Priebe - Profihost AG: > > Hello Yann, > > > > i'm observing some deadlocks again. > > > > I'm using > > httpd 2.4.27 > > + mod_h2 > > +

Re: [Bug 60956] Event MPM listener thread may get blocked by SSL shutdowns

Hello, fullstatus says: Slot PID Stopping ConnectionsThreads Async connections total accepting busy idle writing keep-alive closing 025042 no 0 no2198 0 0 4294966698 14347 no 0 no0200 0

Re: [Bug 60956] Event MPM listener thread may get blocked by SSL shutdowns

Am 19.07.2017 um 16:59 schrieb Stefan Priebe - Profihost AG: > Hello Yann, > > i'm observing some deadlocks again. > > I'm using > httpd 2.4.27 > + mod_h2 > + httpd-2.4.x-mpm_event-wakeup-v7.1.patch > + your ssl linger fix patch from this thread > > What kind of information do you need? If you

Re: [Bug 60956] Event MPM listener thread may get blocked by SSL shutdowns

Hello Yann, i'm observing some deadlocks again. I'm using httpd 2.4.27 + mod_h2 + httpd-2.4.x-mpm_event-wakeup-v7.1.patch + your ssl linger fix patch from this thread What kind of information do you need? If you need a full stack backtrace - from which pid? Or from all httpd pids? Thanks!

Re: [Bug 60956] Event MPM listener thread may get blocked by SSL shutdowns

2017-07-17 9:33 GMT+02:00 Stefan Eissing : > > > Am 14.07.2017 um 21:52 schrieb Yann Ylavic : > > > > On Fri, Jun 30, 2017 at 1:33 PM, Yann Ylavic > wrote: > >> On Fri, Jun 30, 2017 at 1:20 PM, Ruediger Pluem

Re: [Bug 60956] Event MPM listener thread may get blocked by SSL shutdowns

Threw it into my test suite and works nicely. > Am 17.07.2017 um 14:02 schrieb Yann Ylavic : > > On Mon, Jul 17, 2017 at 9:33 AM, Stefan Eissing > wrote: >> >> I will test the patch, most likely today. I lot of +1s for the initiative! > >

Re: [Bug 60956] Event MPM listener thread may get blocked by SSL shutdowns

On Mon, Jul 17, 2017 at 9:33 AM, Stefan Eissing wrote: > > I will test the patch, most likely today. I lot of +1s for the initiative! Thanks Stefan, as I said the proposed patch currently reuses the existing CONN_STATE_LINGER state to shutdown connections, but if it

Re: [Bug 60956] Event MPM listener thread may get blocked by SSL shutdowns

> Am 14.07.2017 um 21:52 schrieb Yann Ylavic : > > On Fri, Jun 30, 2017 at 1:33 PM, Yann Ylavic wrote: >> On Fri, Jun 30, 2017 at 1:20 PM, Ruediger Pluem wrote: >>> >>> On 06/30/2017 12:18 PM, Yann Ylavic wrote: IMHO

Re: [Bug 60956] Event MPM listener thread may get blocked by SSL shutdowns

On Fri, Jun 30, 2017 at 1:33 PM, Yann Ylavic wrote: > On Fri, Jun 30, 2017 at 1:20 PM, Ruediger Pluem wrote: >> >> On 06/30/2017 12:18 PM, Yann Ylavic wrote: >>> >>> IMHO mod_ssl shoudn't (BIO_)flush unconditionally in >>> modssl_smart_shutdown(), only in

Re: [Bug 60956] Event MPM listener thread may get blocked by SSL shutdowns

Hi Yann and Ruediger, 2c from a mpm-event newbie inline: 2017-06-30 13:33 GMT+02:00 Yann Ylavic : > On Fri, Jun 30, 2017 at 1:20 PM, Ruediger Pluem wrote: > > > > On 06/30/2017 12:18 PM, Yann Ylavic wrote: > >> > >> IMHO mod_ssl shoudn't (BIO_)flush

Re: [Bug 60956] Event MPM listener thread may get blocked by SSL shutdowns

> Am 30.06.2017 um 13:33 schrieb Yann Ylavic : > > On Fri, Jun 30, 2017 at 1:20 PM, Ruediger Pluem wrote: >> >> On 06/30/2017 12:18 PM, Yann Ylavic wrote: >>> >>> IMHO mod_ssl shoudn't (BIO_)flush unconditionally in >>> modssl_smart_shutdown(), only in

Re: [Bug 60956] Event MPM listener thread may get blocked by SSL shutdowns

On Fri, Jun 30, 2017 at 1:20 PM, Ruediger Pluem wrote: > > On 06/30/2017 12:18 PM, Yann Ylavic wrote: >> >> IMHO mod_ssl shoudn't (BIO_)flush unconditionally in >> modssl_smart_shutdown(), only in the "abortive" mode of >> ssl_filter_io_shutdown(). > > I think the issue starts

Re: [Bug 60956] Event MPM listener thread may get blocked by SSL shutdowns

On Fri, Jun 30, 2017 at 12:52 PM, Luca Toscano wrote: > > 2017-06-30 12:18 GMT+02:00 Yann Ylavic : >> > >> > http://svn.apache.org/viewvc?view=revision=1706669 >> > http://svn.apache.org/viewvc?view=revision=1734656 >> > >> > IIUC these ones are meant

Re: [Bug 60956] Event MPM listener thread may get blocked by SSL shutdowns

On 06/30/2017 12:18 PM, Yann Ylavic wrote: > Hi Luca, > > [better/easier to talk about details on dev@] > > On Fri, Jun 30, 2017 at 11:05 AM, wrote: >> https://bz.apache.org/bugzilla/show_bug.cgi?id=60956 >> >> --- Comment #11 from Luca Toscano

Re: [Bug 60956] Event MPM listener thread may get blocked by SSL shutdowns

Hi Yann! 2017-06-30 12:18 GMT+02:00 Yann Ylavic : > Hi Luca, > > [better/easier to talk about details on dev@] > > On Fri, Jun 30, 2017 at 11:05 AM, wrote: > > https://bz.apache.org/bugzilla/show_bug.cgi?id=60956 > > > > --- Comment #11 from Luca

Re: [Bug 60956] Event MPM listener thread may get blocked by SSL shutdowns

Hi Luca, [better/easier to talk about details on dev@] On Fri, Jun 30, 2017 at 11:05 AM, wrote: > https://bz.apache.org/bugzilla/show_bug.cgi?id=60956 > > --- Comment #11 from Luca Toscano --- > Other two interesting trunk improvements that have