Re: rabbitmq behind apache / mod_rewrite

[Stefan Priebe - Profihost AG]

Hi

the error Message shows the following URL:
"The requested URL /rabbitmq/api/exchanges// was not found on this server."

Request was: http://XXX/rabbitmq/api/exchanges/%2F

So it seems the %2F gets decoded somewhere even AllowEncodedSlashes  is
set to Yes

Stefan

Am 02.10.19 um 10:33 schrieb Stefan Priebe - Profihost AG:
> Hi,
> 
> i'm trying to run the webgui of rabbitmq behind apache proxy while using
> mod_rewrite.
> 
> RabbitMQ generates URLs like:
> http://rabbit.example.com/#/queues/%2F/myqueue
> 
> but those always generate a 404
> 
> What i tried is:
> AllowEncodedSlashes NoDecode or Yes
> 
> RewriteRule ^/?rabbitmq/(.*)$ http://172.16.206.71:15672/$1 [B=#,NE,P,L]
> 
> or
> 
> RewriteRule ^/?rabbitmq/(.*)$ http://172.16.206.71:15672/$1 [B,PT,L]
> 
> or
> 
> RewriteRule ^/?rabbitmq/(.*)$ http://172.16.206.71:15672/$1 [NE,P,L]
> 
> but all of those and in coombination does not work...
> 
> Can anyhelp help with this?
> 
> 
> This one works fine but i need to use mod_rewrite:
> AllowEncodedSlashes NoDecode
> ProxyPass / http://localhost:15672/ nocanon
> ProxyPassReverse / http://localhost:15672/
> 
> Greets,
> Stefan
> 

rabbitmq behind apache / mod_rewrite

[Stefan Priebe - Profihost AG]

Hi,

i'm trying to run the webgui of rabbitmq behind apache proxy while using
mod_rewrite.

RabbitMQ generates URLs like:
http://rabbit.example.com/#/queues/%2F/myqueue

but those always generate a 404

What i tried is:
AllowEncodedSlashes NoDecode or Yes

RewriteRule ^/?rabbitmq/(.*)$ http://172.16.206.71:15672/$1 [B=#,NE,P,L]

or

RewriteRule ^/?rabbitmq/(.*)$ http://172.16.206.71:15672/$1 [B,PT,L]

or

RewriteRule ^/?rabbitmq/(.*)$ http://172.16.206.71:15672/$1 [NE,P,L]

but all of those and in coombination does not work...

Can anyhelp help with this?


This one works fine but i need to use mod_rewrite:
AllowEncodedSlashes NoDecode
ProxyPass / http://localhost:15672/ nocanon
ProxyPassReverse / http://localhost:15672/

Greets,
Stefan

Apache 0-day / apache-uaf / use after free bugs

[Stefan Priebe - Profihost AG]

Hi,

in twitter and other social media channels they're talking about a
current apache 0 day:
https://twitter.com/i/web/status/1087593706444730369

which wasn't handled / isn't currently fixed.

Some details are here:
https://github.com/hannob/apache-uaf

If this is true there will be exploits soon. Is there anything planned?
Does 2.4.38 fix those issues?

Greets,
Stefan

Re: https vhosts

[Stefan Priebe - Profihost AG]

Hi Stefan,

as i've tried todo nearly the same some weeks ago i can tell you what i did.

Comment inline.

Am 24.05.2018 um 13:34 schrieb Stefan Eissing:
> So, we are lacking an option here to abort SSL connections without a vhost 
> match, it seems. Something like
> 
> SSLStrictSNIVHostCheck require-match

No this will not solve the problem as you may have clients not
supporting sni at all. I tried the same with SSLStrictSNIVHostCheck On

Nowadays those clients see the wrong cert but still land in the correct
vhost container so they "work".

What i did is creating a vhost container which is loaded first in config
order:
1.) Create a fake or general SSL cert (something like a white label cert)
2.) create the container:

  # Because this virtual host is defined first, it will
  # be used as the default if the hostname is not received
  # in the SSL handshake
  ServerName unknown
  DocumentRoot /usr/local/apache/htdocs/

  Protocols h2 http/1.1

  SSLCertificateFile ...
  SSLCertificateKeyFile ...
  SSLCertificateChainFile ...
  SSLEngine on

  ErrorLog /usr/local/apache/logs/htdocs-error.log
  TransferLog /usr/local/apache/logs/htdocs-access.log

  # Rewrite to correct hostname using http - vhost container does not
have https
  RewriteEngine On
  RewriteRule (.*) http://%{HTTP_HOST}/$1 [R=301,L]


Greets,
Stefan

Re: Http2 stats into mod_systemd

[Stefan Priebe - Profihost AG]

Am 04.04.2018 um 11:59 schrieb Stefan Eissing:
> 
>> Am 30.03.2018 um 19:20 schrieb Stefan Priebe - Profihost AG 
>> <s.pri...@profihost.ag>:
>>
>> Hi Stefan,
>>
>> would it be possible to add some http2 stats into mod_systemd status?
> 
> Sorry for the late reply. Just looking at the module and sd_notify() use.
> 
> Since, the status should include data from all child processes, one would
> need to use the scoreboard. And for that, h2 status information would need
> to be added there, AFAIK.
> 
> What exactly would you, as a hypothetical user, be interested in?

mainly what the current status shows as well but for http/2.

* h2 workers
* busy h2 workers
* handled connections

Greets,
Stefan
> 
>>
>> Greets,
>> Stefan
>>
>> Excuse my typo sent from my mobile phone.
> 

Http2 stats into mod_systemd

[Stefan Priebe - Profihost AG]

Hi Stefan,

would it be possible to add some http2 stats into mod_systemd status?

Greets,
Stefan

Excuse my typo sent from my mobile phone.

Re: build mod_systemd static

[Stefan Priebe - Profihost AG]

Am 27.03.2018 um 13:52 schrieb Eric Covener:
> On Tue, Mar 27, 2018 at 7:45 AM, Stefan Priebe - Profihost AG
> <s.pri...@profihost.ag> wrote:
>> Hello,
>>
>> i've backported mod_systemd to httpd 2.4 which was / is very straight
>> forward.
>>
>> Now i would like to build mod_systemd statically but don't get it.
>>
>> While --enable-unixd=static work fine --enable-systemd=static still
>> builds it as a shared module.
>>
>> Any hints?
>>
> I am not too familiar with it, but the pattern used in
> modules/arch/unix/config5.m4 looks a little deviant.
> 
> In the "all prereqs met" case it might be clobbering
> enable_systemd=shared with enable_systemd="yes". I spot checked other
> config.m4's and none seem to do it like that.
> 
> I would try just zapping that line in config5.m4 then re-running configure

Great! That fixes it. I haven't seen that. Would be nice to have this
fixed in trunk as well.

Stefan

build mod_systemd static

[Stefan Priebe - Profihost AG]

Hello,

i've backported mod_systemd to httpd 2.4 which was / is very straight
forward.

Now i would like to build mod_systemd statically but don't get it.

While --enable-unixd=static work fine --enable-systemd=static still
builds it as a shared module.

Any hints?

Greets,
Stefan

Re: Apache server-status Monitoring / why is it only available through http?

[Stefan Priebe - Profihost AG]

might this be something - where mod_systemd can be extended? I have
never used it but it already seems to provide some stats.
Am 26.03.2018 um 12:17 schrieb Stefan Priebe - Profihost AG:
> 
> Am 24.03.2018 um 15:28 schrieb Eric Covener:
>> On Sat, Mar 24, 2018 at 10:18 AM, Stefan Priebe - Profihost AG
>> <s.pri...@profihost.ag> wrote:
>>> Hello,
>>>
>>> is there any reason why the srever-status output isn't:
>>> - available through a local unix socket - so you can grab it even all
>>> workers are busy
>>
>> There are some slightly weird third-party and partial solutions for this
>>
>>  - mod_backdoor (additional listener w/ dedicated thread for
>> emergencies, not AF_UNIX though)
>>  - scripts that separately parse an on-disk "ScoreBoardFile"
> 
> Both sounds like really ugly hacks. ScoreBoardFile isn't recommanded for
> performance reasons (see httpd docs).
> 
> mod_backdoor does not even survive a restart (i didn't test but there is
> a note in the README).
> 
> So both ways do not seem to be really usable.
> 
>> Maybe mod_backdoor with a new name + limitations addressed + more
>> eyeballs would be worth looking at for adding to the distribution.
>> http://people.apache.org/~trawick/mod_backdoor.txt
>>
>>> - available in a machine readable format even it might be csv...
>>
>> There is a machine-readable option in the query string, append ?auto
>> to the URL.
>>
>> -/-
>>
>> Also worth mentioning but I don't really know anything about it is
>> mod_bmx: https://github.com/hyperic/mod_bmx
>>

Re: Apache server-status Monitoring / why is it only available through http?

[Stefan Priebe - Profihost AG]

Am 24.03.2018 um 15:28 schrieb Eric Covener:
> On Sat, Mar 24, 2018 at 10:18 AM, Stefan Priebe - Profihost AG
> <s.pri...@profihost.ag> wrote:
>> Hello,
>>
>> is there any reason why the srever-status output isn't:
>> - available through a local unix socket - so you can grab it even all
>> workers are busy
> 
> There are some slightly weird third-party and partial solutions for this
> 
>  - mod_backdoor (additional listener w/ dedicated thread for
> emergencies, not AF_UNIX though)
>  - scripts that separately parse an on-disk "ScoreBoardFile"

Both sounds like really ugly hacks. ScoreBoardFile isn't recommanded for
performance reasons (see httpd docs).

mod_backdoor does not even survive a restart (i didn't test but there is
a note in the README).

So both ways do not seem to be really usable.

> Maybe mod_backdoor with a new name + limitations addressed + more
> eyeballs would be worth looking at for adding to the distribution.
> http://people.apache.org/~trawick/mod_backdoor.txt
> 
>> - available in a machine readable format even it might be csv...
> 
> There is a machine-readable option in the query string, append ?auto
> to the URL.
> 
> -/-
> 
> Also worth mentioning but I don't really know anything about it is
> mod_bmx: https://github.com/hyperic/mod_bmx
> 

Apache server-status Monitoring / why is it only available through http?

[Stefan Priebe - Profihost AG]

Hello,

is there any reason why the srever-status output isn't:
- available through a local unix socket - so you can grab it even all
workers are busy
- available in a machine readable format even it might be csv...

Thanks!

Greets,
Stefan

Re: apache 2.4.29: mod_http2 stucks from time to time

[Stefan Priebe - Profihost AG]

Hello,

here is the relevant commit:
https://github.com/icing/mod_h2/commit/51085e0c26da1f47ea9cf91930af8cef0dececb9

Stefan

Excuse my typo sent from my mobile phone.

> Am 02.12.2017 um 14:39 schrieb Stefan Priebe - Profihost AG 
> <s.pri...@profihost.ag>:
> 
> Hello Luca,
> 
> It’s a problem with the limited fifo queue size of http2.
> 
> Stefan send me a workaround not sure if this one will go into http2 or if he 
> will release a real fix soon.
> 
> Greets,
> Stefan
> 
> Excuse my typo sent from my mobile phone.
> 
>> Am 02.12.2017 um 09:23 schrieb Luca Toscano <toscano.l...@gmail.com>:
>> 
>> Hi everybody,
>> 
>> did I miss an update or are we still waiting for more data? (Don't mean to 
>> rush you Stefan, just to understand what's the status of the thread :)
>> 
>> Luca
>> 
>> 2017-11-24 15:26 GMT+01:00 Stefan Priebe - Profihost AG 
>> <s.pri...@profihost.ag>:
>>> Thanks i‘ll post a log tonight with a 120s stalled request.
>>> 
>>> Greets,
>>> Stefan
>>> 
>>> Excuse my typo sent from my mobile phone.
>>> 
>>>> Am 23.11.2017 um 17:09 schrieb Stefan Eissing 
>>>> <stefan.eiss...@greenbytes.de>:
>>>> 
>>>> Hey,
>>>> 
>>>> could you try the patch below and produce such a lovely log file again? 
>>>> H2MaxWorkers please back to before, unconfigured. Thanks! This is a small 
>>>> change that a) logs the interaction with h2_workers a bit more and makes 
>>>> sure that time gets lost where I think it does. It also switches the fifo 
>>>> queue in set mode where duplicate entries are checked, in case that 
>>>> interferes here.
>>>> 
>>>> Cheers,
>>>> 
>>>> Stefan
>>>> 
>>>> 
>>>> 
>>>> 
>>>>> Am 23.11.2017 um 14:16 schrieb Stefan Priebe - Profihost AG 
>>>>> <s.pri...@profihost.ag>:
>>>>> 
>>>>> Hi,
>>>>>> Am 23.11.2017 um 14:10 schrieb Stefan Eissing:
>>>>>> Interesting. I assume that otherwise this host is the same (OS/CPU etc.) 
>>>>>> as others where it runs without probs?
>>>>> 
>>>>> Yes and no i got some more reports by colleagues where they've disabled
>>>>> http2 as the customers had unexpected long loading times.
>>>>> 
>>>>>> We are not ghosted by some strange blabla-lake hyper threading thingie 
>>>>>> singularity?
>>>>> 
>>>>> Huhoh what's that? Any chance to add some more debugging?
>>>>> 
>>>>> Greets,
>>>>> Stefan
>>>>> 
>>>>>> 
>>>>>> Need to think about this.
>>>>>> 
>>>>>>> Am 23.11.2017 um 13:43 schrieb Stefan Priebe - Profihost AG 
>>>>>>> <s.pri...@profihost.ag>:
>>>>>>> 
>>>>>>> *argh*, i was too fast no it did NOT fix the problem. It even happens 
>>>>>>> with:
>>>>>>> H2MaxWorkers4096
>>>>>>> 
>>>>>>> Sorry about that.
>>>>>>> 
>>>>>>> Stefan
>>>>>>> 
>>>>>>>> Am 23.11.2017 um 13:42 schrieb Stefan Priebe - Profihost AG:
>>>>>>>> Hello,,
>>>>>>>> 
>>>>>>>> setting:
>>>>>>>> H2MaxWorkers1024
>>>>>>>> 
>>>>>>>> fixes the issue for me. The main problem is how to i know how many
>>>>>>>> workers are needed? How can i detect whether all workers of h2 are 
>>>>>>>> busy?
>>>>>>>> 
>>>>>>>> Stefan
>>>>>>>> 
>>>>>>>>> Am 22.11.2017 um 13:23 schrieb Stefan Priebe - Profihost AG:
>>>>>>>>> Hell Stefan,
>>>>>>>>> 
>>>>>>>>> will send a log to you in a few seconds via private email.
>>>>>>>>> 
>>>>>>>>> Greets,
>>>>>>>>> Stefan
>>>>>>>>> 
>>>>>>>>>> Am 21.11.2017 um 23:18 schrieb Stefan Eissing:
>>>>>>>>>> sorry for the late reply. for stucks trace2 is best. 
>>>>>>>>>> 
>>>>>>

Re: apache 2.4.29: mod_http2 stucks from time to time

[Stefan Priebe - Profihost AG]

Hello Luca,

It’s a problem with the limited fifo queue size of http2.

Stefan send me a workaround not sure if this one will go into http2 or if he 
will release a real fix soon.

Greets,
Stefan

Excuse my typo sent from my mobile phone.

> Am 02.12.2017 um 09:23 schrieb Luca Toscano <toscano.l...@gmail.com>:
> 
> Hi everybody,
> 
> did I miss an update or are we still waiting for more data? (Don't mean to 
> rush you Stefan, just to understand what's the status of the thread :)
> 
> Luca
> 
> 2017-11-24 15:26 GMT+01:00 Stefan Priebe - Profihost AG 
> <s.pri...@profihost.ag>:
>> Thanks i‘ll post a log tonight with a 120s stalled request.
>> 
>> Greets,
>> Stefan
>> 
>> Excuse my typo sent from my mobile phone.
>> 
>>> Am 23.11.2017 um 17:09 schrieb Stefan Eissing 
>>> <stefan.eiss...@greenbytes.de>:
>>> 
>>> Hey,
>>> 
>>> could you try the patch below and produce such a lovely log file again? 
>>> H2MaxWorkers please back to before, unconfigured. Thanks! This is a small 
>>> change that a) logs the interaction with h2_workers a bit more and makes 
>>> sure that time gets lost where I think it does. It also switches the fifo 
>>> queue in set mode where duplicate entries are checked, in case that 
>>> interferes here.
>>> 
>>> Cheers,
>>> 
>>> Stefan
>>> 
>>> 
>>> 
>>> 
>>>> Am 23.11.2017 um 14:16 schrieb Stefan Priebe - Profihost AG 
>>>> <s.pri...@profihost.ag>:
>>>> 
>>>> Hi,
>>>>> Am 23.11.2017 um 14:10 schrieb Stefan Eissing:
>>>>> Interesting. I assume that otherwise this host is the same (OS/CPU etc.) 
>>>>> as others where it runs without probs?
>>>> 
>>>> Yes and no i got some more reports by colleagues where they've disabled
>>>> http2 as the customers had unexpected long loading times.
>>>> 
>>>>> We are not ghosted by some strange blabla-lake hyper threading thingie 
>>>>> singularity?
>>>> 
>>>> Huhoh what's that? Any chance to add some more debugging?
>>>> 
>>>> Greets,
>>>> Stefan
>>>> 
>>>>> 
>>>>> Need to think about this.
>>>>> 
>>>>>> Am 23.11.2017 um 13:43 schrieb Stefan Priebe - Profihost AG 
>>>>>> <s.pri...@profihost.ag>:
>>>>>> 
>>>>>> *argh*, i was too fast no it did NOT fix the problem. It even happens 
>>>>>> with:
>>>>>> H2MaxWorkers4096
>>>>>> 
>>>>>> Sorry about that.
>>>>>> 
>>>>>> Stefan
>>>>>> 
>>>>>>> Am 23.11.2017 um 13:42 schrieb Stefan Priebe - Profihost AG:
>>>>>>> Hello,,
>>>>>>> 
>>>>>>> setting:
>>>>>>> H2MaxWorkers1024
>>>>>>> 
>>>>>>> fixes the issue for me. The main problem is how to i know how many
>>>>>>> workers are needed? How can i detect whether all workers of h2 are busy?
>>>>>>> 
>>>>>>> Stefan
>>>>>>> 
>>>>>>>> Am 22.11.2017 um 13:23 schrieb Stefan Priebe - Profihost AG:
>>>>>>>> Hell Stefan,
>>>>>>>> 
>>>>>>>> will send a log to you in a few seconds via private email.
>>>>>>>> 
>>>>>>>> Greets,
>>>>>>>> Stefan
>>>>>>>> 
>>>>>>>>> Am 21.11.2017 um 23:18 schrieb Stefan Eissing:
>>>>>>>>> sorry for the late reply. for stucks trace2 is best. 
>>>>>>>>> 
>>>>>>>>>> Am 21.11.2017 um 19:35 schrieb Stefan Priebe - Profihost AG 
>>>>>>>>>> <s.pri...@profihost.ag>:
>>>>>>>>>> 
>>>>>>>>>> Hello Stefan,
>>>>>>>>>> 
>>>>>>>>>> which loglevel do you need? trace2?
>>>>>>>>>> 
>>>>>>>>>> Greets,
>>>>>>>>>> Stefan
>>>>>>>>>> 
>>>>>>>>>>> Am 21.11.2017 um 16:48 schrieb Stefan Eissing:
>>>>>>>>>>> Never done this, but https:/

Re: apache 2.4.29: mod_http2 stucks from time to time

[Stefan Priebe - Profihost AG]

Thanks i‘ll post a log tonight with a 120s stalled request.

Greets,
Stefan

Excuse my typo sent from my mobile phone.

> Am 23.11.2017 um 17:09 schrieb Stefan Eissing <stefan.eiss...@greenbytes.de>:
> 
> Hey,
> 
> could you try the patch below and produce such a lovely log file again? 
> H2MaxWorkers please back to before, unconfigured. Thanks! This is a small 
> change that a) logs the interaction with h2_workers a bit more and makes sure 
> that time gets lost where I think it does. It also switches the fifo queue in 
> set mode where duplicate entries are checked, in case that interferes here.
> 
> Cheers,
> 
> Stefan
> 
> 
> 
> 
>> Am 23.11.2017 um 14:16 schrieb Stefan Priebe - Profihost AG 
>> <s.pri...@profihost.ag>:
>> 
>> Hi,
>>> Am 23.11.2017 um 14:10 schrieb Stefan Eissing:
>>> Interesting. I assume that otherwise this host is the same (OS/CPU etc.) as 
>>> others where it runs without probs?
>> 
>> Yes and no i got some more reports by colleagues where they've disabled
>> http2 as the customers had unexpected long loading times.
>> 
>>> We are not ghosted by some strange blabla-lake hyper threading thingie 
>>> singularity?
>> 
>> Huhoh what's that? Any chance to add some more debugging?
>> 
>> Greets,
>> Stefan
>> 
>>> 
>>> Need to think about this.
>>> 
>>>> Am 23.11.2017 um 13:43 schrieb Stefan Priebe - Profihost AG 
>>>> <s.pri...@profihost.ag>:
>>>> 
>>>> *argh*, i was too fast no it did NOT fix the problem. It even happens with:
>>>> H2MaxWorkers4096
>>>> 
>>>> Sorry about that.
>>>> 
>>>> Stefan
>>>> 
>>>>> Am 23.11.2017 um 13:42 schrieb Stefan Priebe - Profihost AG:
>>>>> Hello,,
>>>>> 
>>>>> setting:
>>>>> H2MaxWorkers1024
>>>>> 
>>>>> fixes the issue for me. The main problem is how to i know how many
>>>>> workers are needed? How can i detect whether all workers of h2 are busy?
>>>>> 
>>>>> Stefan
>>>>> 
>>>>>> Am 22.11.2017 um 13:23 schrieb Stefan Priebe - Profihost AG:
>>>>>> Hell Stefan,
>>>>>> 
>>>>>> will send a log to you in a few seconds via private email.
>>>>>> 
>>>>>> Greets,
>>>>>> Stefan
>>>>>> 
>>>>>>> Am 21.11.2017 um 23:18 schrieb Stefan Eissing:
>>>>>>> sorry for the late reply. for stucks trace2 is best. 
>>>>>>> 
>>>>>>>> Am 21.11.2017 um 19:35 schrieb Stefan Priebe - Profihost AG 
>>>>>>>> <s.pri...@profihost.ag>:
>>>>>>>> 
>>>>>>>> Hello Stefan,
>>>>>>>> 
>>>>>>>> which loglevel do you need? trace2?
>>>>>>>> 
>>>>>>>> Greets,
>>>>>>>> Stefan
>>>>>>>> 
>>>>>>>>> Am 21.11.2017 um 16:48 schrieb Stefan Eissing:
>>>>>>>>> Never done this, but https://www.howtoforge.com/setenvif_apache2 
>>>>>>>>> seems like one way to do make it work.
>>>>>>>>> 
>>>>>>>>>> Am 21.11.2017 um 16:16 schrieb Stefan Priebe - Profihost AG 
>>>>>>>>>> <s.pri...@profihost.ag>:
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>>>> Am 21.11.2017 um 16:06 schrieb Stefan Priebe - Profihost AG:
>>>>>>>>>>>>> Am 21.11.2017 um 15:45 schrieb Stefan Eissing:
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Am 21.11.2017 um 14:33 schrieb Stefan Priebe - Profihost AG 
>>>>>>>>>>>>> <s.pri...@profihost.ag>:
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Hello Stefan,
>>>>>>>>>>>>> Hello Yann,
>>>>>>>>>>>>> 
>>>>>>>>>>>>> me the http2 bug tester is calling again ;-)
>>>>>>>>>>>> 
>>>>>>>>>>>> And the day was going so well...
>>>>>>>>>>> 
>>>>>>>>>>> I'm sorry ;-)
>>>>>>>>>>> 
>>>>>>>>>>>>> While running two bash curl while loops the one using http1.1 
>>>>>>>>>>>>> always
>>>>>>>>>>>>> finishes in < 0.05s while the http2 one takes sometimes 0.4 to 
>>>>>>>>>>>>> 20s to
>>>>>>>>>>>>> finish. Sadly i can't reproduce this all the time - mostly more 
>>>>>>>>>>>>> requests
>>>>>>>>>>>>> more failures. As this is a production server i've no idea how to 
>>>>>>>>>>>>> debug
>>>>>>>>>>>>> as the http2 trace logs might flood the harddisk.
>>>>>>>>>>>> 
>>>>>>>>>>>> Hmmm. Do you know if this happens waiting for a response or at the 
>>>>>>>>>>>> end of a connection? Or in the middle of a body? All GETs or also 
>>>>>>>>>>>> POSTs?
>>>>>>>>>>> 
>>>>>>>>>>> My Test only contains GET - but most probably there are also running
>>>>>>>>>>> POST requests but not started by me.
>>>>>>>>>>> 
>>>>>>>>>>> Strangely this only happens between 1pm and 2pm a day but i've no 
>>>>>>>>>>> idea
>>>>>>>>>>> what's different at that time.
>>>>>>>>>> 
>>>>>>>>>> OK i'm also able to reproduce this whenever your want. Can we 
>>>>>>>>>> activate
>>>>>>>>>> trace logging for a specific IP? So i can generate a http2 log?
>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> I can output a lot of information from curl:
>>>>>>>>>>>   time_namelookup
>>>>>>>>>>>   time_connect
>>>>>>>>>>>time_appconnect
>>>>>>>>>>>   time_pretransfer
>>>>>>>>>>>  time_redirect
>>>>>>>>>>> time_starttransfer
>>>>>>>>>>> 
>>>>>>>>>>> Another way might be to enable trace logging only for "my" IP? Is
>>>>>>>>>>> something like this possible?
>>>>>>>>>>> 
>>>>>>>>>>> Greets,
>>>>>>>>>>> Stefan
>>>>>>>>> 
>>>>>>> 
>>> 
> 

Re: apache 2.4.29: mod_http2 stucks from time to time

[Stefan Priebe - Profihost AG]

Hi,
Am 23.11.2017 um 14:10 schrieb Stefan Eissing:
> Interesting. I assume that otherwise this host is the same (OS/CPU etc.) as 
> others where it runs without probs?

Yes and no i got some more reports by colleagues where they've disabled
http2 as the customers had unexpected long loading times.

> We are not ghosted by some strange blabla-lake hyper threading thingie 
> singularity?

Huhoh what's that? Any chance to add some more debugging?

Greets,
Stefan

> 
> Need to think about this.
> 
>> Am 23.11.2017 um 13:43 schrieb Stefan Priebe - Profihost AG 
>> <s.pri...@profihost.ag>:
>>
>> *argh*, i was too fast no it did NOT fix the problem. It even happens with:
>> H2MaxWorkers4096
>>
>> Sorry about that.
>>
>> Stefan
>>
>> Am 23.11.2017 um 13:42 schrieb Stefan Priebe - Profihost AG:
>>> Hello,,
>>>
>>> setting:
>>> H2MaxWorkers1024
>>>
>>> fixes the issue for me. The main problem is how to i know how many
>>> workers are needed? How can i detect whether all workers of h2 are busy?
>>>
>>> Stefan
>>>
>>> Am 22.11.2017 um 13:23 schrieb Stefan Priebe - Profihost AG:
>>>> Hell Stefan,
>>>>
>>>> will send a log to you in a few seconds via private email.
>>>>
>>>> Greets,
>>>> Stefan
>>>>
>>>> Am 21.11.2017 um 23:18 schrieb Stefan Eissing:
>>>>> sorry for the late reply. for stucks trace2 is best. 
>>>>>
>>>>>> Am 21.11.2017 um 19:35 schrieb Stefan Priebe - Profihost AG 
>>>>>> <s.pri...@profihost.ag>:
>>>>>>
>>>>>> Hello Stefan,
>>>>>>
>>>>>> which loglevel do you need? trace2?
>>>>>>
>>>>>> Greets,
>>>>>> Stefan
>>>>>>
>>>>>>> Am 21.11.2017 um 16:48 schrieb Stefan Eissing:
>>>>>>> Never done this, but https://www.howtoforge.com/setenvif_apache2 seems 
>>>>>>> like one way to do make it work.
>>>>>>>
>>>>>>>> Am 21.11.2017 um 16:16 schrieb Stefan Priebe - Profihost AG 
>>>>>>>> <s.pri...@profihost.ag>:
>>>>>>>>
>>>>>>>>
>>>>>>>>> Am 21.11.2017 um 16:06 schrieb Stefan Priebe - Profihost AG:
>>>>>>>>>> Am 21.11.2017 um 15:45 schrieb Stefan Eissing:
>>>>>>>>>>
>>>>>>>>>>> Am 21.11.2017 um 14:33 schrieb Stefan Priebe - Profihost AG 
>>>>>>>>>>> <s.pri...@profihost.ag>:
>>>>>>>>>>>
>>>>>>>>>>> Hello Stefan,
>>>>>>>>>>> Hello Yann,
>>>>>>>>>>>
>>>>>>>>>>> me the http2 bug tester is calling again ;-)
>>>>>>>>>>
>>>>>>>>>> And the day was going so well...
>>>>>>>>>
>>>>>>>>> I'm sorry ;-)
>>>>>>>>>
>>>>>>>>>>> While running two bash curl while loops the one using http1.1 always
>>>>>>>>>>> finishes in < 0.05s while the http2 one takes sometimes 0.4 to 20s 
>>>>>>>>>>> to
>>>>>>>>>>> finish. Sadly i can't reproduce this all the time - mostly more 
>>>>>>>>>>> requests
>>>>>>>>>>> more failures. As this is a production server i've no idea how to 
>>>>>>>>>>> debug
>>>>>>>>>>> as the http2 trace logs might flood the harddisk.
>>>>>>>>>>
>>>>>>>>>> Hmmm. Do you know if this happens waiting for a response or at the 
>>>>>>>>>> end of a connection? Or in the middle of a body? All GETs or also 
>>>>>>>>>> POSTs?
>>>>>>>>>
>>>>>>>>> My Test only contains GET - but most probably there are also running
>>>>>>>>> POST requests but not started by me.
>>>>>>>>>
>>>>>>>>> Strangely this only happens between 1pm and 2pm a day but i've no idea
>>>>>>>>> what's different at that time.
>>>>>>>>
>>>>>>>> OK i'm also able to reproduce this whenever your want. Can we activate
>>>>>>>> trace logging for a specific IP? So i can generate a http2 log?
>>>>>>>>
>>>>>>>>>
>>>>>>>>> I can output a lot of information from curl:
>>>>>>>>> time_namelookup
>>>>>>>>> time_connect
>>>>>>>>>  time_appconnect
>>>>>>>>> time_pretransfer
>>>>>>>>>time_redirect
>>>>>>>>>   time_starttransfer
>>>>>>>>>
>>>>>>>>> Another way might be to enable trace logging only for "my" IP? Is
>>>>>>>>> something like this possible?
>>>>>>>>>
>>>>>>>>> Greets,
>>>>>>>>> Stefan
>>>>>>>
>>>>>
> 

Re: apache 2.4.29: mod_http2 stucks from time to time

[Stefan Priebe - Profihost AG]

*argh*, i was too fast no it did NOT fix the problem. It even happens with:
H2MaxWorkers4096

Sorry about that.

Stefan

Am 23.11.2017 um 13:42 schrieb Stefan Priebe - Profihost AG:
> Hello,,
> 
> setting:
> H2MaxWorkers1024
> 
> fixes the issue for me. The main problem is how to i know how many
> workers are needed? How can i detect whether all workers of h2 are busy?
> 
> Stefan
> 
> Am 22.11.2017 um 13:23 schrieb Stefan Priebe - Profihost AG:
>> Hell Stefan,
>>
>> will send a log to you in a few seconds via private email.
>>
>> Greets,
>> Stefan
>>
>> Am 21.11.2017 um 23:18 schrieb Stefan Eissing:
>>> sorry for the late reply. for stucks trace2 is best. 
>>>
>>>> Am 21.11.2017 um 19:35 schrieb Stefan Priebe - Profihost AG 
>>>> <s.pri...@profihost.ag>:
>>>>
>>>> Hello Stefan,
>>>>
>>>> which loglevel do you need? trace2?
>>>>
>>>> Greets,
>>>> Stefan
>>>>
>>>>> Am 21.11.2017 um 16:48 schrieb Stefan Eissing:
>>>>> Never done this, but https://www.howtoforge.com/setenvif_apache2 seems 
>>>>> like one way to do make it work.
>>>>>
>>>>>> Am 21.11.2017 um 16:16 schrieb Stefan Priebe - Profihost AG 
>>>>>> <s.pri...@profihost.ag>:
>>>>>>
>>>>>>
>>>>>>> Am 21.11.2017 um 16:06 schrieb Stefan Priebe - Profihost AG:
>>>>>>>> Am 21.11.2017 um 15:45 schrieb Stefan Eissing:
>>>>>>>>
>>>>>>>>> Am 21.11.2017 um 14:33 schrieb Stefan Priebe - Profihost AG 
>>>>>>>>> <s.pri...@profihost.ag>:
>>>>>>>>>
>>>>>>>>> Hello Stefan,
>>>>>>>>> Hello Yann,
>>>>>>>>>
>>>>>>>>> me the http2 bug tester is calling again ;-)
>>>>>>>>
>>>>>>>> And the day was going so well...
>>>>>>>
>>>>>>> I'm sorry ;-)
>>>>>>>
>>>>>>>>> While running two bash curl while loops the one using http1.1 always
>>>>>>>>> finishes in < 0.05s while the http2 one takes sometimes 0.4 to 20s to
>>>>>>>>> finish. Sadly i can't reproduce this all the time - mostly more 
>>>>>>>>> requests
>>>>>>>>> more failures. As this is a production server i've no idea how to 
>>>>>>>>> debug
>>>>>>>>> as the http2 trace logs might flood the harddisk.
>>>>>>>>
>>>>>>>> Hmmm. Do you know if this happens waiting for a response or at the end 
>>>>>>>> of a connection? Or in the middle of a body? All GETs or also POSTs?
>>>>>>>
>>>>>>> My Test only contains GET - but most probably there are also running
>>>>>>> POST requests but not started by me.
>>>>>>>
>>>>>>> Strangely this only happens between 1pm and 2pm a day but i've no idea
>>>>>>> what's different at that time.
>>>>>>
>>>>>> OK i'm also able to reproduce this whenever your want. Can we activate
>>>>>> trace logging for a specific IP? So i can generate a http2 log?
>>>>>>
>>>>>>>
>>>>>>> I can output a lot of information from curl:
>>>>>>>  time_namelookup
>>>>>>>  time_connect
>>>>>>>   time_appconnect
>>>>>>>  time_pretransfer
>>>>>>> time_redirect
>>>>>>>time_starttransfer
>>>>>>>
>>>>>>> Another way might be to enable trace logging only for "my" IP? Is
>>>>>>> something like this possible?
>>>>>>>
>>>>>>> Greets,
>>>>>>> Stefan
>>>>>
>>>

Re: apache 2.4.29: mod_http2 stucks from time to time

[Stefan Priebe - Profihost AG]

Hello,,

setting:
H2MaxWorkers1024

fixes the issue for me. The main problem is how to i know how many
workers are needed? How can i detect whether all workers of h2 are busy?

Stefan

Am 22.11.2017 um 13:23 schrieb Stefan Priebe - Profihost AG:
> Hell Stefan,
> 
> will send a log to you in a few seconds via private email.
> 
> Greets,
> Stefan
> 
> Am 21.11.2017 um 23:18 schrieb Stefan Eissing:
>> sorry for the late reply. for stucks trace2 is best. 
>>
>>> Am 21.11.2017 um 19:35 schrieb Stefan Priebe - Profihost AG 
>>> <s.pri...@profihost.ag>:
>>>
>>> Hello Stefan,
>>>
>>> which loglevel do you need? trace2?
>>>
>>> Greets,
>>> Stefan
>>>
>>>> Am 21.11.2017 um 16:48 schrieb Stefan Eissing:
>>>> Never done this, but https://www.howtoforge.com/setenvif_apache2 seems 
>>>> like one way to do make it work.
>>>>
>>>>> Am 21.11.2017 um 16:16 schrieb Stefan Priebe - Profihost AG 
>>>>> <s.pri...@profihost.ag>:
>>>>>
>>>>>
>>>>>> Am 21.11.2017 um 16:06 schrieb Stefan Priebe - Profihost AG:
>>>>>>> Am 21.11.2017 um 15:45 schrieb Stefan Eissing:
>>>>>>>
>>>>>>>> Am 21.11.2017 um 14:33 schrieb Stefan Priebe - Profihost AG 
>>>>>>>> <s.pri...@profihost.ag>:
>>>>>>>>
>>>>>>>> Hello Stefan,
>>>>>>>> Hello Yann,
>>>>>>>>
>>>>>>>> me the http2 bug tester is calling again ;-)
>>>>>>>
>>>>>>> And the day was going so well...
>>>>>>
>>>>>> I'm sorry ;-)
>>>>>>
>>>>>>>> While running two bash curl while loops the one using http1.1 always
>>>>>>>> finishes in < 0.05s while the http2 one takes sometimes 0.4 to 20s to
>>>>>>>> finish. Sadly i can't reproduce this all the time - mostly more 
>>>>>>>> requests
>>>>>>>> more failures. As this is a production server i've no idea how to debug
>>>>>>>> as the http2 trace logs might flood the harddisk.
>>>>>>>
>>>>>>> Hmmm. Do you know if this happens waiting for a response or at the end 
>>>>>>> of a connection? Or in the middle of a body? All GETs or also POSTs?
>>>>>>
>>>>>> My Test only contains GET - but most probably there are also running
>>>>>> POST requests but not started by me.
>>>>>>
>>>>>> Strangely this only happens between 1pm and 2pm a day but i've no idea
>>>>>> what's different at that time.
>>>>>
>>>>> OK i'm also able to reproduce this whenever your want. Can we activate
>>>>> trace logging for a specific IP? So i can generate a http2 log?
>>>>>
>>>>>>
>>>>>> I can output a lot of information from curl:
>>>>>>  time_namelookup
>>>>>>  time_connect
>>>>>>   time_appconnect
>>>>>>  time_pretransfer
>>>>>> time_redirect
>>>>>>time_starttransfer
>>>>>>
>>>>>> Another way might be to enable trace logging only for "my" IP? Is
>>>>>> something like this possible?
>>>>>>
>>>>>> Greets,
>>>>>> Stefan
>>>>
>>

Re: apache 2.4.29: mod_http2 stucks from time to time

[Stefan Priebe - Profihost AG]

Hell Stefan,

will send a log to you in a few seconds via private email.

Greets,
Stefan

Am 21.11.2017 um 23:18 schrieb Stefan Eissing:
> sorry for the late reply. for stucks trace2 is best. 
> 
>> Am 21.11.2017 um 19:35 schrieb Stefan Priebe - Profihost AG 
>> <s.pri...@profihost.ag>:
>>
>> Hello Stefan,
>>
>> which loglevel do you need? trace2?
>>
>> Greets,
>> Stefan
>>
>>> Am 21.11.2017 um 16:48 schrieb Stefan Eissing:
>>> Never done this, but https://www.howtoforge.com/setenvif_apache2 seems like 
>>> one way to do make it work.
>>>
>>>> Am 21.11.2017 um 16:16 schrieb Stefan Priebe - Profihost AG 
>>>> <s.pri...@profihost.ag>:
>>>>
>>>>
>>>>> Am 21.11.2017 um 16:06 schrieb Stefan Priebe - Profihost AG:
>>>>>> Am 21.11.2017 um 15:45 schrieb Stefan Eissing:
>>>>>>
>>>>>>> Am 21.11.2017 um 14:33 schrieb Stefan Priebe - Profihost AG 
>>>>>>> <s.pri...@profihost.ag>:
>>>>>>>
>>>>>>> Hello Stefan,
>>>>>>> Hello Yann,
>>>>>>>
>>>>>>> me the http2 bug tester is calling again ;-)
>>>>>>
>>>>>> And the day was going so well...
>>>>>
>>>>> I'm sorry ;-)
>>>>>
>>>>>>> While running two bash curl while loops the one using http1.1 always
>>>>>>> finishes in < 0.05s while the http2 one takes sometimes 0.4 to 20s to
>>>>>>> finish. Sadly i can't reproduce this all the time - mostly more requests
>>>>>>> more failures. As this is a production server i've no idea how to debug
>>>>>>> as the http2 trace logs might flood the harddisk.
>>>>>>
>>>>>> Hmmm. Do you know if this happens waiting for a response or at the end 
>>>>>> of a connection? Or in the middle of a body? All GETs or also POSTs?
>>>>>
>>>>> My Test only contains GET - but most probably there are also running
>>>>> POST requests but not started by me.
>>>>>
>>>>> Strangely this only happens between 1pm and 2pm a day but i've no idea
>>>>> what's different at that time.
>>>>
>>>> OK i'm also able to reproduce this whenever your want. Can we activate
>>>> trace logging for a specific IP? So i can generate a http2 log?
>>>>
>>>>>
>>>>> I can output a lot of information from curl:
>>>>>  time_namelookup
>>>>>  time_connect
>>>>>   time_appconnect
>>>>>  time_pretransfer
>>>>> time_redirect
>>>>>time_starttransfer
>>>>>
>>>>> Another way might be to enable trace logging only for "my" IP? Is
>>>>> something like this possible?
>>>>>
>>>>> Greets,
>>>>> Stefan
>>>
> 

Re: apache 2.4.29: mod_http2 stucks from time to time

[Stefan Priebe - Profihost AG]

Hello Stefan,

which loglevel do you need? trace2?

Greets,
Stefan

Am 21.11.2017 um 16:48 schrieb Stefan Eissing:
> Never done this, but https://www.howtoforge.com/setenvif_apache2 seems like 
> one way to do make it work.
> 
>> Am 21.11.2017 um 16:16 schrieb Stefan Priebe - Profihost AG 
>> <s.pri...@profihost.ag>:
>>
>>
>> Am 21.11.2017 um 16:06 schrieb Stefan Priebe - Profihost AG:
>>> Am 21.11.2017 um 15:45 schrieb Stefan Eissing:
>>>>
>>>>> Am 21.11.2017 um 14:33 schrieb Stefan Priebe - Profihost AG 
>>>>> <s.pri...@profihost.ag>:
>>>>>
>>>>> Hello Stefan,
>>>>> Hello Yann,
>>>>>
>>>>> me the http2 bug tester is calling again ;-)
>>>>
>>>> And the day was going so well...
>>>
>>> I'm sorry ;-)
>>>
>>>>> While running two bash curl while loops the one using http1.1 always
>>>>> finishes in < 0.05s while the http2 one takes sometimes 0.4 to 20s to
>>>>> finish. Sadly i can't reproduce this all the time - mostly more requests
>>>>> more failures. As this is a production server i've no idea how to debug
>>>>> as the http2 trace logs might flood the harddisk.
>>>>
>>>> Hmmm. Do you know if this happens waiting for a response or at the end of 
>>>> a connection? Or in the middle of a body? All GETs or also POSTs?
>>>
>>> My Test only contains GET - but most probably there are also running
>>> POST requests but not started by me.
>>>
>>> Strangely this only happens between 1pm and 2pm a day but i've no idea
>>> what's different at that time.
>>
>> OK i'm also able to reproduce this whenever your want. Can we activate
>> trace logging for a specific IP? So i can generate a http2 log?
>>
>>>
>>> I can output a lot of information from curl:
>>>   time_namelookup
>>>   time_connect
>>>time_appconnect
>>>   time_pretransfer
>>>  time_redirect
>>> time_starttransfer
>>>
>>> Another way might be to enable trace logging only for "my" IP? Is
>>> something like this possible?
>>>
>>> Greets,
>>> Stefan
> 

Re: apache 2.4.29: mod_http2 stucks from time to time

[Stefan Priebe - Profihost AG]

Am 21.11.2017 um 16:06 schrieb Stefan Priebe - Profihost AG:
> Am 21.11.2017 um 15:45 schrieb Stefan Eissing:
>>
>>> Am 21.11.2017 um 14:33 schrieb Stefan Priebe - Profihost AG 
>>> <s.pri...@profihost.ag>:
>>>
>>> Hello Stefan,
>>>  Hello Yann,
>>>
>>> me the http2 bug tester is calling again ;-)
>>
>> And the day was going so well...
> 
> I'm sorry ;-)
> 
>>> While running two bash curl while loops the one using http1.1 always
>>> finishes in < 0.05s while the http2 one takes sometimes 0.4 to 20s to
>>> finish. Sadly i can't reproduce this all the time - mostly more requests
>>> more failures. As this is a production server i've no idea how to debug
>>> as the http2 trace logs might flood the harddisk.
>>
>> Hmmm. Do you know if this happens waiting for a response or at the end of a 
>> connection? Or in the middle of a body? All GETs or also POSTs?
> 
> My Test only contains GET - but most probably there are also running
> POST requests but not started by me.
> 
> Strangely this only happens between 1pm and 2pm a day but i've no idea
> what's different at that time.

OK i'm also able to reproduce this whenever your want. Can we activate
trace logging for a specific IP? So i can generate a http2 log?

> 
> I can output a lot of information from curl:
>time_namelookup
>time_connect
> time_appconnect
>time_pretransfer
>   time_redirect
>  time_starttransfer
> 
> Another way might be to enable trace logging only for "my" IP? Is
> something like this possible?
> 
> Greets,
> Stefan
> 

Re: apache 2.4.29: mod_http2 stucks from time to time

[Stefan Priebe - Profihost AG]

Am 21.11.2017 um 15:45 schrieb Stefan Eissing:
> 
>> Am 21.11.2017 um 14:33 schrieb Stefan Priebe - Profihost AG 
>> <s.pri...@profihost.ag>:
>>
>> Hello Stefan,
>>  Hello Yann,
>>
>> me the http2 bug tester is calling again ;-)
> 
> And the day was going so well...

I'm sorry ;-)

>> While running two bash curl while loops the one using http1.1 always
>> finishes in < 0.05s while the http2 one takes sometimes 0.4 to 20s to
>> finish. Sadly i can't reproduce this all the time - mostly more requests
>> more failures. As this is a production server i've no idea how to debug
>> as the http2 trace logs might flood the harddisk.
> 
> Hmmm. Do you know if this happens waiting for a response or at the end of a 
> connection? Or in the middle of a body? All GETs or also POSTs?

My Test only contains GET - but most probably there are also running
POST requests but not started by me.

Strangely this only happens between 1pm and 2pm a day but i've no idea
what's different at that time.

I can output a lot of information from curl:
   time_namelookup
   time_connect
time_appconnect
   time_pretransfer
  time_redirect
 time_starttransfer

Another way might be to enable trace logging only for "my" IP? Is
something like this possible?

Greets,
Stefan

apache 2.4.29: mod_http2 stucks from time to time

[Stefan Priebe - Profihost AG]

Hello Stefan,
  Hello Yann,

me the http2 bug tester is calling again ;-)

We've the problem "most probably" since apadhe 2.4.29 that http2 stucks
from time to time.

While running two bash curl while loops the one using http1.1 always
finishes in < 0.05s while the http2 one takes sometimes 0.4 to 20s to
finish. Sadly i can't reproduce this all the time - mostly more requests
more failures. As this is a production server i've no idea how to debug
as the http2 trace logs might flood the harddisk.

Thanks!

Greets,
Stefan

Re: We have soon 5 SVN repo's

[Stefan Priebe - Profihost AG]

> Am 04.11.2017 um 11:03 schrieb Steffen :
> 
> Soon we have:
> 
> branches 2.4.x
> trunk
> 2.5.0-alpha
> patches/2.4.x
> patches/trunk
> 
> Please a procedure:  where and when do we apply patches/fixes. 
> 
Why don‘t move to git?

Greets,
Stefan

Excuse my typo sent from my mobile phone.


> 
> 
> 

Re: Time for 2.4.28 ?

[Stefan Priebe - Profihost AG]

> Am 19.09.2017 um 15:50 schrieb Jacob Perkins :
> 
> 
> 
>> On Sep 18, 2017, at 11:24 AM, Jim Jagielski  wrote:
>> 
>> Seems like a good time to push for a 2.4.28 T by end-of-week.
>> Review STATUS! Test and Vote on backports!
>> 
>> Comments?

Do we get this one into 2.4.28?

> https://bz.apache.org/bugzilla/show_bug.cgi?id=60956

Greets,
Stefan

implement multiple AuthType's

[Stefan Priebe - Profihost AG]

Hello,

would it be possible to implement multiple AuthType's? I know that this
is not possible in all cases but it would be great to be able have a
fallback.

For example:
Having GSSAPI Auth in the first place and if the browser does not send a
valid ticket fallback to other auth systems like ldap.

Greets,
Stefan

Re: [Bug 60956] Event MPM listener thread may get blocked by SSL shutdowns

[Stefan Priebe - Profihost AG]

4tr i was able to fix this by mod_h2 v1.10.10

Greets,
Stefan

Am 25.07.2017 um 15:40 schrieb Stefan Eissing:
> Well, if the customer could reproduce this at a 
> 
>   LogLevel http2:trace2
> 
> that would help.
> 
>> Am 25.07.2017 um 15:38 schrieb Stefan Priebe - Profihost AG 
>> <s.pri...@profihost.ag>:
>>
>> Hello Stefan,
>>
>> thanks for the patch. No it does not solve the problem our customer is
>> seeing.
>>
>> What kind of details / logs you need?
>>
>> Greets,
>> Stefan
>>
>> Am 25.07.2017 um 11:59 schrieb Stefan Eissing:
>>> The issue was opened here: https://github.com/icing/mod_h2/issues/143
>>>
>>> I made a patch that i hope addresses the problem. The 2.4.x version I 
>>> attach to this mail.
>>>
>>> Thanks!
>>>
>>> Stefan
>>>
>>>
>>>
>>>
>>>
>>>> Am 25.07.2017 um 08:13 schrieb Stefan Priebe - Profihost AG 
>>>> <s.pri...@profihost.ag>:
>>>>
>>>>
>>>> Am 24.07.2017 um 23:06 schrieb Stefan Eissing:
>>>>> I have another report of request getting stuck, resulting in the error 
>>>>> you noticed. Will look tomorrow and report back here what I find.
>>>>
>>>> Thanks, if you need any logs. Pleae ask.
>>>>
>>>> Stefan
>>>>
>>>>>
>>>>>> Am 24.07.2017 um 22:20 schrieb Stefan Priebe - Profihost AG 
>>>>>> <s.pri...@profihost.ag>:
>>>>>>
>>>>>> Hello all,
>>>>>>
>>>>>> currently 8 hours of testing without any issues.
>>>>>>
>>>>>> @Stefan
>>>>>> i've most probably another issue with http2 where some elements of the
>>>>>> page are sometimes missing and the connection results in
>>>>>> ERR_CONNECTION_CLOSED after 60s. What kind of details do you need?
>>>>>>
>>>>>> Greets,
>>>>>> Stefan
>>>>>>> Am 22.07.2017 um 13:35 schrieb Yann Ylavic:
>>>>>>>> On Sat, Jul 22, 2017 at 2:18 AM, Yann Ylavic <ylavic@gmail.com> 
>>>>>>>> wrote:
>>>>>>>> On Fri, Jul 21, 2017 at 10:31 PM, Stefan Priebe - Profihost AG
>>>>>>>> <s.pri...@profihost.ag> wrote:
>>>>>>>>>
>>>>>>>>> your new defer linger V3 deadlocked as well.
>>>>>>>>>
>>>>>>>>> GDB traces:
>>>>>>>>> https://www.apaste.info/LMfJ
>>>>>>>>
>>>>>>>> This shows the listener thread waiting for a worker while there are
>>>>>>>> many available.
>>>>>>>> My mistake, the worker threads failed to rearm their idle state for
>>>>>>>> the deferred close case.
>>>>>>>>
>>>>>>>> V4 available, thanks Stefan!
>>>>>>>
>>>>>>> Since I didn't want you to test again something that fails quickly, I
>>>>>>> did some stress tests myself this time with several tools, with V5
>>>>>>> (same as V4 from a 2.4.x POV).
>>>>>>>
>>>>>>> It seems to work well here...
>>>>>>>
>>>>>>>> Regards,
>>>>>>>> Yann.
>>>>>
>>>
> 

Re: [Bug 60956] Event MPM listener thread may get blocked by SSL shutdowns

[Stefan Priebe - Profihost AG]

Hello Stefan,

thanks for the patch. No it does not solve the problem our customer is
seeing.

What kind of details / logs you need?

Greets,
Stefan

Am 25.07.2017 um 11:59 schrieb Stefan Eissing:
> The issue was opened here: https://github.com/icing/mod_h2/issues/143
> 
> I made a patch that i hope addresses the problem. The 2.4.x version I attach 
> to this mail.
> 
> Thanks!
> 
> Stefan
> 
> 
> 
> 
> 
>> Am 25.07.2017 um 08:13 schrieb Stefan Priebe - Profihost AG 
>> <s.pri...@profihost.ag>:
>>
>>
>> Am 24.07.2017 um 23:06 schrieb Stefan Eissing:
>>> I have another report of request getting stuck, resulting in the error you 
>>> noticed. Will look tomorrow and report back here what I find.
>>
>> Thanks, if you need any logs. Pleae ask.
>>
>> Stefan
>>
>>>
>>>> Am 24.07.2017 um 22:20 schrieb Stefan Priebe - Profihost AG 
>>>> <s.pri...@profihost.ag>:
>>>>
>>>> Hello all,
>>>>
>>>> currently 8 hours of testing without any issues.
>>>>
>>>> @Stefan
>>>> i've most probably another issue with http2 where some elements of the
>>>> page are sometimes missing and the connection results in
>>>> ERR_CONNECTION_CLOSED after 60s. What kind of details do you need?
>>>>
>>>> Greets,
>>>> Stefan
>>>>> Am 22.07.2017 um 13:35 schrieb Yann Ylavic:
>>>>>> On Sat, Jul 22, 2017 at 2:18 AM, Yann Ylavic <ylavic@gmail.com> 
>>>>>> wrote:
>>>>>> On Fri, Jul 21, 2017 at 10:31 PM, Stefan Priebe - Profihost AG
>>>>>> <s.pri...@profihost.ag> wrote:
>>>>>>>
>>>>>>> your new defer linger V3 deadlocked as well.
>>>>>>>
>>>>>>> GDB traces:
>>>>>>> https://www.apaste.info/LMfJ
>>>>>>
>>>>>> This shows the listener thread waiting for a worker while there are
>>>>>> many available.
>>>>>> My mistake, the worker threads failed to rearm their idle state for
>>>>>> the deferred close case.
>>>>>>
>>>>>> V4 available, thanks Stefan!
>>>>>
>>>>> Since I didn't want you to test again something that fails quickly, I
>>>>> did some stress tests myself this time with several tools, with V5
>>>>> (same as V4 from a 2.4.x POV).
>>>>>
>>>>> It seems to work well here...
>>>>>
>>>>>> Regards,
>>>>>> Yann.
>>>
> 

Re: [Bug 60956] Event MPM listener thread may get blocked by SSL shutdowns

[Stefan Priebe - Profihost AG]

Am 24.07.2017 um 23:06 schrieb Stefan Eissing:
> I have another report of request getting stuck, resulting in the error you 
> noticed. Will look tomorrow and report back here what I find.

Thanks, if you need any logs. Pleae ask.

Stefan

> 
>> Am 24.07.2017 um 22:20 schrieb Stefan Priebe - Profihost AG 
>> <s.pri...@profihost.ag>:
>>
>> Hello all,
>>
>> currently 8 hours of testing without any issues.
>>
>> @Stefan
>> i've most probably another issue with http2 where some elements of the
>> page are sometimes missing and the connection results in
>> ERR_CONNECTION_CLOSED after 60s. What kind of details do you need?
>>
>> Greets,
>> Stefan
>>> Am 22.07.2017 um 13:35 schrieb Yann Ylavic:
>>>> On Sat, Jul 22, 2017 at 2:18 AM, Yann Ylavic <ylavic@gmail.com> wrote:
>>>> On Fri, Jul 21, 2017 at 10:31 PM, Stefan Priebe - Profihost AG
>>>> <s.pri...@profihost.ag> wrote:
>>>>>
>>>>> your new defer linger V3 deadlocked as well.
>>>>>
>>>>> GDB traces:
>>>>> https://www.apaste.info/LMfJ
>>>>
>>>> This shows the listener thread waiting for a worker while there are
>>>> many available.
>>>> My mistake, the worker threads failed to rearm their idle state for
>>>> the deferred close case.
>>>>
>>>> V4 available, thanks Stefan!
>>>
>>> Since I didn't want you to test again something that fails quickly, I
>>> did some stress tests myself this time with several tools, with V5
>>> (same as V4 from a 2.4.x POV).
>>>
>>> It seems to work well here...
>>>
>>>> Regards,
>>>> Yann.
> 

Re: [Bug 60956] Event MPM listener thread may get blocked by SSL shutdowns

[Stefan Priebe - Profihost AG]

Hello all,

currently 8 hours of testing without any issues.

@Stefan
i've most probably another issue with http2 where some elements of the
page are sometimes missing and the connection results in
ERR_CONNECTION_CLOSED after 60s. What kind of details do you need?

Greets,
Stefan
Am 22.07.2017 um 13:35 schrieb Yann Ylavic:
> On Sat, Jul 22, 2017 at 2:18 AM, Yann Ylavic <ylavic@gmail.com> wrote:
>> On Fri, Jul 21, 2017 at 10:31 PM, Stefan Priebe - Profihost AG
>> <s.pri...@profihost.ag> wrote:
>>>
>>> your new defer linger V3 deadlocked as well.
>>>
>>> GDB traces:
>>> https://www.apaste.info/LMfJ
>>
>> This shows the listener thread waiting for a worker while there are
>> many available.
>> My mistake, the worker threads failed to rearm their idle state for
>> the deferred close case.
>>
>> V4 available, thanks Stefan!
> 
> Since I didn't want you to test again something that fails quickly, I
> did some stress tests myself this time with several tools, with V5
> (same as V4 from a 2.4.x POV).
> 
> It seems to work well here...
> 
>> Regards,
>> Yann.

Re: [Bug 60956] Event MPM listener thread may get blocked by SSL shutdowns

[Stefan Priebe - Profihost AG]

First test with version five looks good so far will continue extensive testing 
tomorrow.

Greets,
Stefan

Excuse my typo sent from my mobile phone.

> Am 22.07.2017 um 13:35 schrieb Yann Ylavic <ylavic@gmail.com>:
> 
>> On Sat, Jul 22, 2017 at 2:18 AM, Yann Ylavic <ylavic@gmail.com> wrote:
>> On Fri, Jul 21, 2017 at 10:31 PM, Stefan Priebe - Profihost AG
>> <s.pri...@profihost.ag> wrote:
>>> 
>>> your new defer linger V3 deadlocked as well.
>>> 
>>> GDB traces:
>>> https://www.apaste.info/LMfJ
>> 
>> This shows the listener thread waiting for a worker while there are
>> many available.
>> My mistake, the worker threads failed to rearm their idle state for
>> the deferred close case.
>> 
>> V4 available, thanks Stefan!
> 
> Since I didn't want you to test again something that fails quickly, I
> did some stress tests myself this time with several tools, with V5
> (same as V4 from a 2.4.x POV).
> 
> It seems to work well here...
> 
>> Regards,
>> Yann.

Re: [Bug 60956] Event MPM listener thread may get blocked by SSL shutdowns

[Stefan Priebe - Profihost AG]

Hello Yann,

your new defer linger V3 deadlocked as well.

GDB traces:
https://www.apaste.info/LMfJ

But this time i have no fullstatus for you as the apache didn't serve
any connections at all anymore. But even before i did NOT see those
strange values for closing connections.

Thanks!

Greets,
Stefan
Am 21.07.2017 um 01:08 schrieb Yann Ylavic:
> On Thu, Jul 20, 2017 at 12:48 PM, Stefan Priebe - Profihost AG
> <s.pri...@profihost.ag> wrote:
>> V3 didn't help.
> 
> I just posted a new patch in this thread, with a new approach which I
> think is better anyway.
> 
> Would you mind testing it in your environment?
> 
> 
> Regards,
> Yann.
> 

Re: [Bug 60956] Event MPM listener thread may get blocked by SSL shutdowns

[Stefan Priebe - Profihost AG]

Hello Yann,

i downloaded V3. Can't guarantee when i can test. May be today or on monday.

Greets,
Stefan

Am 21.07.2017 um 01:08 schrieb Yann Ylavic:
> On Thu, Jul 20, 2017 at 12:48 PM, Stefan Priebe - Profihost AG
> <s.pri...@profihost.ag> wrote:
>> V3 didn't help.
> 
> I just posted a new patch in this thread, with a new approach which I
> think is better anyway.
> 
> Would you mind testing it in your environment?
> 
> 
> Regards,
> Yann.
> 

Re: [Bug 60956] Event MPM listener thread may get blocked by SSL shutdowns

[Stefan Priebe - Profihost AG]

Yes:
Slot  PID  Stopping   ConnectionsThreads   Async connections
  total accepting busy idle writing keep-alive  closing
  03614  no   1 no4196  0   0
4294966701
  13615  no   0 no5195  0   0
4294966697
  210228 no   0 no6194  0   0
4294966698
  312030 no   0 no4196  0   0
4294966697
  419495 no   3 yes   6194  0   0
4294966801
  522098 no   6 yes   5195  0   5
4294966706
  630071 no   15yes   8192  0   5
4294967283
  Sum  7 025  38   1362 0   10 -3489

Stefan

Excuse my typo sent from my mobile phone.

> Am 20.07.2017 um 15:18 schrieb Yann Ylavic <ylavic@gmail.com>:
> 
> On Thu, Jul 20, 2017 at 2:58 PM, Stefan Priebe - Profihost AG
> <s.pri...@profihost.ag> wrote:
>> Yes it looks the same but I can't tell if it is.
>> 
>> Here's a backtrace from V3:
>> https://apaste.info/Aw0r
> 
> Thanks Stefan, how about mod_status, still some strange entries?
> 
> Regards,
> Yann.

Re: [Bug 60956] Event MPM listener thread may get blocked by SSL shutdowns

[Stefan Priebe - Profihost AG]

Yes it looks the same but I can't tell if it is.

Here's a backtrace from V3:
https://apaste.info/Aw0r

Greets,
Stefan

Excuse my typo sent from my mobile phone.

> Am 20.07.2017 um 13:01 schrieb Yann Ylavic <ylavic@gmail.com>:
> 
> On Thu, Jul 20, 2017 at 12:48 PM, Stefan Priebe - Profihost AG
> <s.pri...@profihost.ag> wrote:
>> V3 didn't help. Will post a new gdb backtrace soon takes some time as I'm on
>> holiday.
> 
> Thanks Stefan, still some/the same issue in status?
> 
> Regards,
> Yann.

Re: [Bug 60956] Event MPM listener thread may get blocked by SSL shutdowns

[Stefan Priebe - Profihost AG]

V3 didn't help. Will post a new gdb backtrace soon takes some time as I'm on 
holiday.

Stefan

Excuse my typo sent from my mobile phone.

> Am 20.07.2017 um 01:26 schrieb Yann Ylavic <ylavic@gmail.com>:
> 
> On Wed, Jul 19, 2017 at 11:14 PM, Stefan Priebe - Profihost AG
> <s.pri...@profihost.ag> wrote:
>> Am 19.07.2017 um 22:46 schrieb Yann Ylavic:
>>> 
>>> Attached is a v2 if you feel confident enough, still ;)
>> 
>> Thanks, yes i will.
> 
> If you managed to install v2 already you may want to ignore this new
> v3, which only addresses a very unlikely error case where
> lingering_count++ is missing (plus some non-functional changes, a bit
> of renaming and the factorization which would have avoided this
> mistake in the first place).
> 
> Otherwise, you could try this one instead.
> 
> Thanks,
> Yann.
> 

Re: [Bug 60956] Event MPM listener thread may get blocked by SSL shutdowns

[Stefan Priebe - Profihost AG]

Am 20.07.2017 um 01:26 schrieb Yann Ylavic:
> On Wed, Jul 19, 2017 at 11:14 PM, Stefan Priebe - Profihost AG
> <s.pri...@profihost.ag> wrote:
>> Am 19.07.2017 um 22:46 schrieb Yann Ylavic:
>>>
>>> Attached is a v2 if you feel confident enough, still ;)
>>
>> Thanks, yes i will.
> 
> If you managed to install v2 already you may want to ignore this new
> v3, which only addresses a very unlikely error case where
> lingering_count++ is missing (plus some non-functional changes, a bit
> of renaming and the factorization which would have avoided this
> mistake in the first place).
> 
> Otherwise, you could try this one instead.

Thanks, switched to V3.

> 
> Thanks,
> Yann.
> 

Re: [Bug 60956] Event MPM listener thread may get blocked by SSL shutdowns

[Stefan Priebe - Profihost AG]

Am 19.07.2017 um 22:46 schrieb Yann Ylavic:
> Hi Stefan,
> 
> thanks for testing again!
> 
> On Wed, Jul 19, 2017 at 7:42 PM, Stefan Priebe - Profihost AG
> <s.pri...@profihost.ag> wrote:
>>
>> What looks strange
>> from a first view is that async connections closing has very high and
>> strange values:
>> 4294967211
> 
> Indeed, I messed up with mpm_event's lingering_count in the first patch.
> And it can lead to disabling the listener, which I think is what you observe.
> 
> Attached is a v2 if you feel confident enough, still ;)

Thanks, yes i will.

> 
> Regards,
> Yann.
> 

Re: [Bug 60956] Event MPM listener thread may get blocked by SSL shutdowns

[Stefan Priebe - Profihost AG]

Hello,

here we go:

This one is from a server where the first httpd process got stuck:

   Slot  PID  Stopping   ConnectionsThreads   Async connections
   total accepting busy idle writing keep-alive  closing
   031675 no   0 no0200  0   0
4294966700

gdb thread apply all from this pid:
https://apaste.info/cBT5

Greets,
Stefan

Am 19.07.2017 um 17:48 schrieb Luca Toscano:
> Hello Stefan,
> 
> 2017-07-19 17:05 GMT+02:00 Stefan Priebe - Profihost AGlo
> <s.pri...@profihost.ag <mailto:s.pri...@profihost.ag>>:
> 
> 
>     Am 19.07.2017 um 16:59 schrieb Stefan Priebe - Profihost AG:
> > Hello Yann,
> >
> > i'm observing some deadlocks again.
> >
> > I'm using
> > httpd 2.4.27
> > + mod_h2
> > + httpd-2.4.x-mpm_event-wakeup-v7.1.patch
> > + your ssl linger fix patch from this thread
> >
> > What kind of information do you need? If you need a full stack backtrace
> >  - from which pid? Or from all httpd pids?
> 
> Something i forgot to tell:
> 
> it seems httpd is running at max threads:
> awk '{print $10 $11}' lsof.txt | sort | uniq -c | grep LISTEN
>   25050 *:http(LISTEN)
>   25050 *:https(LISTEN)
> 
> 
> First of all let me tell you how awesome is your regular testing, thank
> you! It is helping a ton to deliver stable code :)
> 
> From my point of view I think that you can attach gdb to one or more
> httpd processes and do the usual "thread apply all", IIUC your httpd
> ends up having all of its processes in more or less the same state right?
> 
> Let's use https://apaste.info/ or similar though otherwise we'll need to
> exchange super long emails.
> 
> Thanks again!
> 
> Luca  

Re: [Bug 60956] Event MPM listener thread may get blocked by SSL shutdowns

[Stefan Priebe - Profihost AG]

Hello Luca,

i need to wait until a machine is crashing again. What looks strange
from a first view is that async connections closing has very high and
strange values:
4294967211

Even a not yet crashed system has those:

   Slot  PID  Stopping   ConnectionsThreads   Async connections
   total accepting busy idle writing keep-alive  closing
   025157 no   12yes   4196  0   9
4294967231 <== HERE ==
   125159 no   22yes   8192  0   13
4294967211 <== HERE ==
   Sum  2 034  12   388  0   22 -150

Greets,
Stefan

Am 19.07.2017 um 17:48 schrieb Luca Toscano:
> Hello Stefan,
> 
> 2017-07-19 17:05 GMT+02:00 Stefan Priebe - Profihost AG
> <s.pri...@profihost.ag <mailto:s.pri...@profihost.ag>>:
> 
> 
>     Am 19.07.2017 um 16:59 schrieb Stefan Priebe - Profihost AG:
> > Hello Yann,
> >
> > i'm observing some deadlocks again.
> >
> > I'm using
> > httpd 2.4.27
> > + mod_h2
> > + httpd-2.4.x-mpm_event-wakeup-v7.1.patch
> > + your ssl linger fix patch from this thread
> >
> > What kind of information do you need? If you need a full stack backtrace
> >  - from which pid? Or from all httpd pids?
> 
> Something i forgot to tell:
> 
> it seems httpd is running at max threads:
> awk '{print $10 $11}' lsof.txt | sort | uniq -c | grep LISTEN
>   25050 *:http(LISTEN)
>   25050 *:https(LISTEN)
> 
> 
> First of all let me tell you how awesome is your regular testing, thank
> you! It is helping a ton to deliver stable code :)
> 
> From my point of view I think that you can attach gdb to one or more
> httpd processes and do the usual "thread apply all", IIUC your httpd
> ends up having all of its processes in more or less the same state right?
> 
> Let's use https://apaste.info/ or similar though otherwise we'll need to
> exchange super long emails.
> 
> Thanks again!
> 
> Luca  

Re: [Bug 60956] Event MPM listener thread may get blocked by SSL shutdowns

[Stefan Priebe - Profihost AG]

Hello,

fullstatus says:
   Slot  PID  Stopping   ConnectionsThreads   Async connections
   total accepting busy idle writing keep-alive  closing
   025042 no   0 no2198  0   0
4294966698
   14347  no   0 no0200  0   0
4294966700
   226273 no   0 no1199  0   0
4294966698
   34348  no   0 no0200  0   0
4294966699
   410224 no   0 no0200  0   0
4294966697
   512157 no   0 no0200  0   0
4294966700
   623027 no   0 no0200  0   0
4294966698
   728597 no   0 no0200  0   0
4294966698
   87519  no   0 no0200  0   0
4294966697
   918609 no   0 no2198  0   0
4294966698
   10   3183  no   0 no0200  0   0
4294966698
   11   14704 no   0 no0200  0   0
4294966698
   12   26237 no   0 no0200  0   0
4294966700
   13   32070 no   0 no0200  0   0
4294966697
   14   12070 no   1 no0200  0   0
4294966699
   15   16627 no   0 no0200  0   0
4294966698
   16   29413 no   0 no0200  0   0
4294966699
   17   435   no   0 no0200  0   0
4294966699
   18   24808 no   0 no0200  0   0
4294966700
   19   1822  no   0 no0200  0   0
4294966699
   20   1721  no   0 no0200  0   0
4294966698
   21   2875  no   0 no0200  0   0
4294966698
   22   25879 no   0 no0200  0   0
4294966698
   23   28091 no   0 no0200  0   0
4294966696
   24   31452 no   0 no0200  0   0
4294966698
   25   32706 no   0 no0200  0   0
4294966698
   26   8858  no   14yes   3197  0   6
4294966783
   27   10203 no   5 yes   2198  0   2
4294966949
   Sum  28020  10   5590 0   8  -16400

Greets,
Stefan

Am 19.07.2017 um 17:05 schrieb Stefan Priebe - Profihost AG:
> 
> Am 19.07.2017 um 16:59 schrieb Stefan Priebe - Profihost AG:
>> Hello Yann,
>>
>> i'm observing some deadlocks again.
>>
>> I'm using
>> httpd 2.4.27
>> + mod_h2
>> + httpd-2.4.x-mpm_event-wakeup-v7.1.patch
>> + your ssl linger fix patch from this thread
>>
>> What kind of information do you need? If you need a full stack backtrace
>>  - from which pid? Or from all httpd pids?
> 
> Something i forgot to tell:
> 
> it seems httpd is running at max threads:
> awk '{print $10 $11}' lsof.txt | sort | uniq -c | grep LISTEN
>   25050 *:http(LISTEN)
>   25050 *:https(LISTEN)
> 
> Stefan
> 
>>
>> Thanks!
>>
>> Greets,
>> Stefan
>>
>> Am 14.07.2017 um 21:52 schrieb Yann Ylavic:
>>> On Fri, Jun 30, 2017 at 1:33 PM, Yann Ylavic <ylavic@gmail.com> wrote:
>>>> On Fri, Jun 30, 2017 at 1:20 PM, Ruediger Pluem <rpl...@apache.org> wrote:
>>>>>
>>>>> On 06/30/2017 12:18 PM, Yann Ylavic wrote:
>>>>>>
>>>>>> IMHO mod_ssl shoudn't (BIO_)flush unconditionally in
>>>>>> modssl_smart_shutdown(), only in the "abortive" mode of
>>>>>> ssl_filter_io_shutdown().
>>>>>
>>>>> I think the issue starts before that.
>>>>> ap_prep_lingering_close calls the pre_close_connection hook and modules 
>>>>> that are registered
>>>>> to this hook can perform all sort of long lasting blocking operations 
>>>>> there.
>>>>> While it can be argued that this would be a bug in the module I think the 
>>>>> only safe way
>>>>> is to have the whole start_lingering_close_nonblocking being executed by 
>>>>> a worker thread.
>>>>
>>>> Correct, that'd be much simpler/safer indeed.
>>>> We need a new SHUTDOWN state then, right?
>>>
>>> Actually it was less simple than expected, and it has some caveats 
>>> obviously...
>>>
>>> The attached patch does not introduce a new state but reuses the
>>> existing CONN_STATE_LINGER since it was not really considered by the
>>> listener thread (which uses CONN_STATE_LINGER_NORMAL and
>>> CONN_STATE_LINGER_SHORT instead), but that's a detail.
>>>
>>> Mainly, start_lingering_close_nonblocking() now simply schedules a
>>> shutdown (i.e. pre_close_connection() followed by immediate close)
>>> 

Re: [Bug 60956] Event MPM listener thread may get blocked by SSL shutdowns

[Stefan Priebe - Profihost AG]

Am 19.07.2017 um 16:59 schrieb Stefan Priebe - Profihost AG:
> Hello Yann,
> 
> i'm observing some deadlocks again.
> 
> I'm using
> httpd 2.4.27
> + mod_h2
> + httpd-2.4.x-mpm_event-wakeup-v7.1.patch
> + your ssl linger fix patch from this thread
> 
> What kind of information do you need? If you need a full stack backtrace
>  - from which pid? Or from all httpd pids?

Something i forgot to tell:

it seems httpd is running at max threads:
awk '{print $10 $11}' lsof.txt | sort | uniq -c | grep LISTEN
  25050 *:http(LISTEN)
  25050 *:https(LISTEN)

Stefan

> 
> Thanks!
> 
> Greets,
> Stefan
> 
> Am 14.07.2017 um 21:52 schrieb Yann Ylavic:
>> On Fri, Jun 30, 2017 at 1:33 PM, Yann Ylavic <ylavic@gmail.com> wrote:
>>> On Fri, Jun 30, 2017 at 1:20 PM, Ruediger Pluem <rpl...@apache.org> wrote:
>>>>
>>>> On 06/30/2017 12:18 PM, Yann Ylavic wrote:
>>>>>
>>>>> IMHO mod_ssl shoudn't (BIO_)flush unconditionally in
>>>>> modssl_smart_shutdown(), only in the "abortive" mode of
>>>>> ssl_filter_io_shutdown().
>>>>
>>>> I think the issue starts before that.
>>>> ap_prep_lingering_close calls the pre_close_connection hook and modules 
>>>> that are registered
>>>> to this hook can perform all sort of long lasting blocking operations 
>>>> there.
>>>> While it can be argued that this would be a bug in the module I think the 
>>>> only safe way
>>>> is to have the whole start_lingering_close_nonblocking being executed by a 
>>>> worker thread.
>>>
>>> Correct, that'd be much simpler/safer indeed.
>>> We need a new SHUTDOWN state then, right?
>>
>> Actually it was less simple than expected, and it has some caveats 
>> obviously...
>>
>> The attached patch does not introduce a new state but reuses the
>> existing CONN_STATE_LINGER since it was not really considered by the
>> listener thread (which uses CONN_STATE_LINGER_NORMAL and
>> CONN_STATE_LINGER_SHORT instead), but that's a detail.
>>
>> Mainly, start_lingering_close_nonblocking() now simply schedules a
>> shutdown (i.e. pre_close_connection() followed by immediate close)
>> that will we be run by a worker thread.
>> A new shutdown_linger_q is created/handled (with the same timeout as
>> the short_linger_q, namely 2 seconds) to hold connections to be
>> shutdown.
>>
>> So now when a connection times out in the write_completion or
>> keepalive queues, it needs (i.e. the listener may wait for) an
>> available worker to process its shutdown/close.
>> This means we can *not* close kept alive connections immediatly like
>> before when becoming short of workers, which will favor active KA
>> connections over new ones in this case (I don't think it's that
>> serious but the previous was taking care of that. For me it's up to
>> the admin to size the workers appropriately...).
>>
>> Same when a connection in the shutdown_linger_q itself times out, the
>> patch will require a worker immediatly to do the job (see
>> shutdown_lingering_close() callback).
>>
>> So overall, this patch may introduce the need for more workers than
>> before, what was (wrongly) done by the listener thread has to be done
>> somewhere anyway...
>>
>> Finally, I think there is room for improvements like batching
>> shutdowns in the same worker if there is no objection on the approach
>> so far.
>>
>> WDYT?
>>
>> Regards,
>> Yann.
>>

Re: [Bug 60956] Event MPM listener thread may get blocked by SSL shutdowns

[Stefan Priebe - Profihost AG]

Hello Yann,

i'm observing some deadlocks again.

I'm using
httpd 2.4.27
+ mod_h2
+ httpd-2.4.x-mpm_event-wakeup-v7.1.patch
+ your ssl linger fix patch from this thread

What kind of information do you need? If you need a full stack backtrace
 - from which pid? Or from all httpd pids?

Thanks!

Greets,
Stefan

Am 14.07.2017 um 21:52 schrieb Yann Ylavic:
> On Fri, Jun 30, 2017 at 1:33 PM, Yann Ylavic  wrote:
>> On Fri, Jun 30, 2017 at 1:20 PM, Ruediger Pluem  wrote:
>>>
>>> On 06/30/2017 12:18 PM, Yann Ylavic wrote:

 IMHO mod_ssl shoudn't (BIO_)flush unconditionally in
 modssl_smart_shutdown(), only in the "abortive" mode of
 ssl_filter_io_shutdown().
>>>
>>> I think the issue starts before that.
>>> ap_prep_lingering_close calls the pre_close_connection hook and modules 
>>> that are registered
>>> to this hook can perform all sort of long lasting blocking operations there.
>>> While it can be argued that this would be a bug in the module I think the 
>>> only safe way
>>> is to have the whole start_lingering_close_nonblocking being executed by a 
>>> worker thread.
>>
>> Correct, that'd be much simpler/safer indeed.
>> We need a new SHUTDOWN state then, right?
> 
> Actually it was less simple than expected, and it has some caveats 
> obviously...
> 
> The attached patch does not introduce a new state but reuses the
> existing CONN_STATE_LINGER since it was not really considered by the
> listener thread (which uses CONN_STATE_LINGER_NORMAL and
> CONN_STATE_LINGER_SHORT instead), but that's a detail.
> 
> Mainly, start_lingering_close_nonblocking() now simply schedules a
> shutdown (i.e. pre_close_connection() followed by immediate close)
> that will we be run by a worker thread.
> A new shutdown_linger_q is created/handled (with the same timeout as
> the short_linger_q, namely 2 seconds) to hold connections to be
> shutdown.
> 
> So now when a connection times out in the write_completion or
> keepalive queues, it needs (i.e. the listener may wait for) an
> available worker to process its shutdown/close.
> This means we can *not* close kept alive connections immediatly like
> before when becoming short of workers, which will favor active KA
> connections over new ones in this case (I don't think it's that
> serious but the previous was taking care of that. For me it's up to
> the admin to size the workers appropriately...).
> 
> Same when a connection in the shutdown_linger_q itself times out, the
> patch will require a worker immediatly to do the job (see
> shutdown_lingering_close() callback).
> 
> So overall, this patch may introduce the need for more workers than
> before, what was (wrongly) done by the listener thread has to be done
> somewhere anyway...
> 
> Finally, I think there is room for improvements like batching
> shutdowns in the same worker if there is no objection on the approach
> so far.
> 
> WDYT?
> 
> Regards,
> Yann.
> 

Apache 2.4.26 mpm_event_listener_wakeup_bug57399_V7

[Stefan Priebe - Profihost AG]

Hello Yann,

is your wakeup patch (mpm_event_listener_wakeup_bug57399_V7) still
needed for 2.4.26?

Greets,
Stefan

Re: The drive for 2.4.26

[Stefan Priebe - Profihost AG]

Hi Luca,

Am 27.05.2017 um 09:10 schrieb Luca Toscano:
> Hi Jim,
> 
> 2017-05-25 14:23 GMT+02:00 Jim Jagielski  >:
> 
> Until a proposal in in STATUS (and apologies if it is), its
> suitability for 2.4.x is a big unknown.
> 
> 
> you are completely right, afaik the last patch tested by Stefan Priebe
> was https://bz.apache.org/bugzilla/attachment.cgi?id=34641=diff
> (Stefan correct me if I am wrong).

Yes that's right. Yann told something about a V8 coming soon but that's
weeks ago. V7 seems to run fine. Currently running on nearly 2000 vms.

> If I remember correctly it might be possible that some commits were not
> necessary since they were made as a test to remove segfaults that we
> discovered due to other things. I'll wait Yann's feedback but if anybody
> wants to check the patch it would be really great.
> 
> Thanks!
> 
> Luca 

VUDDY: unpatched CVEs in apache httpd

[Stefan Priebe - Profihost AG]

Hello list,

while reading "http://www.ieee-security.org/TC/SP2017/papers/71.pdf;
they claim to have found unpatched security holes in apache httpd. While
reading further it seems that the only missing peace is the unpatched
xmlparse from expat.

While searching on our build server it turns out to me that the bug is
not in apache httpd but in aprutil:

# grep -nHr 'newHash = hash' apr-util-1.5.4

apr-util-1.5.4/xml/expat/lib/xmlparse.c:5431:  unsigned long
newHash = hash(table->v[i]->name);

Did i miss anything from the paper? Is a new apr-util version planned
which fixes the problem? Are there any special build options or modules
needed?

Greets,
Stefan

Re: hanging apache httpd processes due to mod_h2

[Stefan Priebe - Profihost AG]

looks good so war. Will wait some more days.

Greets,
Stefan

Am 19.04.2017 um 11:09 schrieb Stefan Eissing:
> Thanks, the backtraces really helped. Could you try the following patch?
> 
> -Stefan
> 
> 
> 
> 
>> Am 19.04.2017 um 08:32 schrieb Stefan Priebe - Profihost AG 
>> <s.pri...@profihost.ag>:
>>
>> Hi Stefan,
>>
>> this is now with v1.10.2 but it does not help.
>>
>> PID old gen:
>>
>> 215608   yes (old gen)   2   no  0   0   0   0   >> 0
>>
>>
>> requests in G state:
>>
>> 2-2  15608   4/36/36 G   252.75  30695   41  22.10.870.87
>> A.B.C.D h2  XYZ:443
>> GET /themes/Frontend/Responsive/frontend/_public/src/js/vendors
>>
>> 2-2  15608   2/5/5   G   69.62   39645   36  104.0   0.180.18
>> A.B.C.D h2  XYZ:443 GET
>> /web/cache/1492526985_9e591f3c4f420ebca36f3c1be30bd5ee.css
>>
>> thread all bt:
>> https://pastebin.com/raw/Cq9ymt9u
>>
>> I think the interesting thread is:
>> Thread 6
>>
>> Greets,
>> Stefan
>>
>> Am 18.04.2017 um 15:06 schrieb Stefan Priebe - Profihost AG:
>>>
>>> Am 18.04.2017 um 15:03 schrieb Stefan Eissing:
>>>> Stefan,
>>>>
>>>> that is a 1.10.0, right? That was the first version without nested locking 
>>>> and I fixed 2 possible dead locks in 1.10.1. 
>>>>
>>>> I am about to release a 1.10.2 with added conformity checks and a fix for 
>>>> client omitting EOF flags. Could you give that one a try?
>>>
>>> Yes sure where is 1.10.2?
>>>
>>> Stefan
>>>
>>>>
>>>> -Stefan
>>>>
>>>>> Am 18.04.2017 um 14:57 schrieb Stefan Priebe - Profihost AG 
>>>>> <s.pri...@profihost.ag>:
>>>>>
>>>>> Hi,
>>>>>
>>>>> i saw that all of them are still serving one h2 connection.
>>>>>
>>>>> server-status:
>>>>> 0-3   32375   42/64/181   G   30.09   1020776 214 1285.1  
>>>>> 2.405.81
>>>>> h081217236127.dyn.cm.kabsi.at h2  :443GET
>>>>> /wp-content/uploads/Bloglr21-8003asd.jpg HTTP/2.0
>>>>>
>>>>> And they all have Mode of operation => G
>>>>>
>>>>> thread apply all bt full shows:
>>>>> #0  0x7f8e80b687fc in __lll_lock_wait () from
>>>>> /lib/x86_64-linux-gnu/libpthread.so.0
>>>>> No symbol table info available.
>>>>> #1  0x7f8e80b6b6f8 in _L_cond_lock_886 () from
>>>>> /lib/x86_64-linux-gnu/libpthread.so.0
>>>>> No symbol table info available.
>>>>> #2  0x7f8e80b6b464 in __pthread_mutex_cond_lock () from
>>>>> /lib/x86_64-linux-gnu/libpthread.so.0
>>>>> No symbol table info available.
>>>>> #3  0x7f8e80b6650a in pthread_cond_timedwait@@GLIBC_2.3.2 () from
>>>>> /lib/x86_64-linux-gnu/libpthread.so.0
>>>>> No symbol table info available.
>>>>> #4  0x7f8e80d92322 in apr_thread_cond_timedwait
>>>>> (cond=0x7f8d7846bc70, mutex=0x7f8d7846bc20, timeout=6000) at
>>>>> locks/unix/thread_cond.c:89
>>>>>   rv = 
>>>>>   then = 
>>>>>   abstime = {tv_sec = 1491843095, tv_nsec = 700198000}
>>>>> #5  0x55e62c0951c5 in r_wait_space (premain=,
>>>>> pbl=0x7f8de853da60, block=APR_BLOCK_READ, beam=0x7f8d7846bb08) at
>>>>> h2_bucket_beam.c:337
>>>>>   status = 
>>>>> #6  append_bucket (pbl=0x7f8de853da60, block=APR_BLOCK_READ,
>>>>> b=0x7f8d7846dd38, beam=0x7f8d7846bb08) at h2_bucket_beam.c:776
>>>>>   data = 0x7f8de853dac8 "\240\037"
>>>>>   len = 0
>>>>>   space_left = 0
>>>>>   status = 
>>>>> #7  h2_beam_send (beam=0x7f8d7846bb08, sender_bb=0x7f8d7864ba90,
>>>>> block=APR_BLOCK_READ) at h2_bucket_beam.c:906
>>>>>   force_report = 1
>>>>>   b = 0x7f8d7846dd38
>>>>>   status = 0
>>>>>   bl = {mutex = 0x7f8d7846bc20, leave = 0x55e62c094250
>>>>> , leave_ctx = 0x0}
>>>>> #8  0x55e62c08f3aa in send_out (task=0x7f8d7846f740,
>>>>> bb=0x7f8d7864ba90, block=1) at h2_task.c:100
>>>>>   written = 8096
>>>

Re: hanging apache httpd processes due to mod_h2

[Stefan Priebe - Profihost AG]

Hi Stefan,

this is now with v1.10.2 but it does not help.

PID old gen:

2   15608   yes (old gen)   2   no  0   0   0   0   0


requests in G state:

2-2 15608   4/36/36 G   252.75  30695   41  22.10.870.87
A.B.C.D h2  XYZ:443
GET /themes/Frontend/Responsive/frontend/_public/src/js/vendors

2-2 15608   2/5/5   G   69.62   39645   36  104.0   0.180.18
A.B.C.D h2  XYZ:443 GET
/web/cache/1492526985_9e591f3c4f420ebca36f3c1be30bd5ee.css

thread all bt:
https://pastebin.com/raw/Cq9ymt9u

I think the interesting thread is:
Thread 6

Greets,
Stefan

Am 18.04.2017 um 15:06 schrieb Stefan Priebe - Profihost AG:
> 
> Am 18.04.2017 um 15:03 schrieb Stefan Eissing:
>> Stefan,
>>
>> that is a 1.10.0, right? That was the first version without nested locking 
>> and I fixed 2 possible dead locks in 1.10.1. 
>>
>> I am about to release a 1.10.2 with added conformity checks and a fix for 
>> client omitting EOF flags. Could you give that one a try?
> 
> Yes sure where is 1.10.2?
> 
> Stefan
> 
>>
>> -Stefan
>>
>>> Am 18.04.2017 um 14:57 schrieb Stefan Priebe - Profihost AG 
>>> <s.pri...@profihost.ag>:
>>>
>>> Hi,
>>>
>>> i saw that all of them are still serving one h2 connection.
>>>
>>> server-status:
>>> 0-3 32375   42/64/181   G   30.09   1020776 214 1285.1  2.40
>>> 5.81
>>> h081217236127.dyn.cm.kabsi.at   h2  :443GET
>>> /wp-content/uploads/Bloglr21-8003asd.jpg HTTP/2.0
>>>
>>> And they all have Mode of operation => G
>>>
>>> thread apply all bt full shows:
>>> #0  0x7f8e80b687fc in __lll_lock_wait () from
>>> /lib/x86_64-linux-gnu/libpthread.so.0
>>> No symbol table info available.
>>> #1  0x7f8e80b6b6f8 in _L_cond_lock_886 () from
>>> /lib/x86_64-linux-gnu/libpthread.so.0
>>> No symbol table info available.
>>> #2  0x7f8e80b6b464 in __pthread_mutex_cond_lock () from
>>> /lib/x86_64-linux-gnu/libpthread.so.0
>>> No symbol table info available.
>>> #3  0x7f8e80b6650a in pthread_cond_timedwait@@GLIBC_2.3.2 () from
>>> /lib/x86_64-linux-gnu/libpthread.so.0
>>> No symbol table info available.
>>> #4  0x7f8e80d92322 in apr_thread_cond_timedwait
>>> (cond=0x7f8d7846bc70, mutex=0x7f8d7846bc20, timeout=6000) at
>>> locks/unix/thread_cond.c:89
>>>rv = 
>>>then = 
>>>abstime = {tv_sec = 1491843095, tv_nsec = 700198000}
>>> #5  0x55e62c0951c5 in r_wait_space (premain=,
>>> pbl=0x7f8de853da60, block=APR_BLOCK_READ, beam=0x7f8d7846bb08) at
>>> h2_bucket_beam.c:337
>>>status = 
>>> #6  append_bucket (pbl=0x7f8de853da60, block=APR_BLOCK_READ,
>>> b=0x7f8d7846dd38, beam=0x7f8d7846bb08) at h2_bucket_beam.c:776
>>>data = 0x7f8de853dac8 "\240\037"
>>>len = 0
>>>space_left = 0
>>>status = 
>>> #7  h2_beam_send (beam=0x7f8d7846bb08, sender_bb=0x7f8d7864ba90,
>>> block=APR_BLOCK_READ) at h2_bucket_beam.c:906
>>>force_report = 1
>>>b = 0x7f8d7846dd38
>>>status = 0
>>>bl = {mutex = 0x7f8d7846bc20, leave = 0x55e62c094250
>>> , leave_ctx = 0x0}
>>> #8  0x55e62c08f3aa in send_out (task=0x7f8d7846f740,
>>> bb=0x7f8d7864ba90, block=1) at h2_task.c:100
>>>written = 8096
>>>left = 140245585033024
>>>status = 1
>>> #9  0x55e62c08f976 in slave_out (task=0x7f8d7846f740,
>>> f=0x7f8d7846bdd8, bb=0x7f8d7864ba90) at h2_task.c:177
>>>status = -512
>>>flush = 0
>>>blocking = 1
>>> #10 0x55e62c08fbfd in h2_filter_slave_output (filter=0x7f8d7846bdd8,
>>> brigade=0x7f8d7864ba90) at h2_task.c:370
>>>task = 0x7f8d7846f740
>>>status = 
>>> #11 0x55e62bffc01c in ap_content_length_filter (f=0x7f8d78472ca8,
>>> b=0x7f8d7864ba90) at protocol.c:1713
>>>r = 0x7f8d78471750
>>>ctx = 0x7f8d7864bc90
>>>e = 0x7f8d7864ba98
>>>eblock = (unknown: 2159446012)
>>> #12 0x55e62c04853d in deflate_out_filter (f=0x7f8d7864b650,
>>> bb=0x7f8d7864b8f8) at mod_deflate.c:961
>>>rv = -512
>>>r = 0x7f8d78471750
>>>ctx = 0x7f8d7864b9a8
>>>len = 8096
>>>blen = 1157663
>>>data = 0x7f

Re: hanging apache httpd processes

[Stefan Priebe - Profihost AG]

Am 18.04.2017 um 15:03 schrieb Stefan Eissing:
> Stefan,
> 
> that is a 1.10.0, right? That was the first version without nested locking 
> and I fixed 2 possible dead locks in 1.10.1. 
> 
> I am about to release a 1.10.2 with added conformity checks and a fix for 
> client omitting EOF flags. Could you give that one a try?

Yes sure where is 1.10.2?

Stefan

> 
> -Stefan
> 
>> Am 18.04.2017 um 14:57 schrieb Stefan Priebe - Profihost AG 
>> <s.pri...@profihost.ag>:
>>
>> Hi,
>>
>> i saw that all of them are still serving one h2 connection.
>>
>> server-status:
>> 0-3  32375   42/64/181   G   30.09   1020776 214 1285.1  2.40
>> 5.81
>> h081217236127.dyn.cm.kabsi.ath2  :443GET
>> /wp-content/uploads/Bloglr21-8003asd.jpg HTTP/2.0
>>
>> And they all have Mode of operation => G
>>
>> thread apply all bt full shows:
>> #0  0x7f8e80b687fc in __lll_lock_wait () from
>> /lib/x86_64-linux-gnu/libpthread.so.0
>> No symbol table info available.
>> #1  0x7f8e80b6b6f8 in _L_cond_lock_886 () from
>> /lib/x86_64-linux-gnu/libpthread.so.0
>> No symbol table info available.
>> #2  0x7f8e80b6b464 in __pthread_mutex_cond_lock () from
>> /lib/x86_64-linux-gnu/libpthread.so.0
>> No symbol table info available.
>> #3  0x7f8e80b6650a in pthread_cond_timedwait@@GLIBC_2.3.2 () from
>> /lib/x86_64-linux-gnu/libpthread.so.0
>> No symbol table info available.
>> #4  0x7f8e80d92322 in apr_thread_cond_timedwait
>> (cond=0x7f8d7846bc70, mutex=0x7f8d7846bc20, timeout=6000) at
>> locks/unix/thread_cond.c:89
>>rv = 
>>then = 
>>abstime = {tv_sec = 1491843095, tv_nsec = 700198000}
>> #5  0x55e62c0951c5 in r_wait_space (premain=,
>> pbl=0x7f8de853da60, block=APR_BLOCK_READ, beam=0x7f8d7846bb08) at
>> h2_bucket_beam.c:337
>>status = 
>> #6  append_bucket (pbl=0x7f8de853da60, block=APR_BLOCK_READ,
>> b=0x7f8d7846dd38, beam=0x7f8d7846bb08) at h2_bucket_beam.c:776
>>data = 0x7f8de853dac8 "\240\037"
>>len = 0
>>space_left = 0
>>status = 
>> #7  h2_beam_send (beam=0x7f8d7846bb08, sender_bb=0x7f8d7864ba90,
>> block=APR_BLOCK_READ) at h2_bucket_beam.c:906
>>force_report = 1
>>b = 0x7f8d7846dd38
>>status = 0
>>bl = {mutex = 0x7f8d7846bc20, leave = 0x55e62c094250
>> , leave_ctx = 0x0}
>> #8  0x55e62c08f3aa in send_out (task=0x7f8d7846f740,
>> bb=0x7f8d7864ba90, block=1) at h2_task.c:100
>>written = 8096
>>left = 140245585033024
>>status = 1
>> #9  0x55e62c08f976 in slave_out (task=0x7f8d7846f740,
>> f=0x7f8d7846bdd8, bb=0x7f8d7864ba90) at h2_task.c:177
>>status = -512
>>flush = 0
>>blocking = 1
>> #10 0x55e62c08fbfd in h2_filter_slave_output (filter=0x7f8d7846bdd8,
>> brigade=0x7f8d7864ba90) at h2_task.c:370
>>task = 0x7f8d7846f740
>>status = 
>> #11 0x55e62bffc01c in ap_content_length_filter (f=0x7f8d78472ca8,
>> b=0x7f8d7864ba90) at protocol.c:1713
>>r = 0x7f8d78471750
>>ctx = 0x7f8d7864bc90
>>e = 0x7f8d7864ba98
>>eblock = (unknown: 2159446012)
>> #12 0x55e62c04853d in deflate_out_filter (f=0x7f8d7864b650,
>> bb=0x7f8d7864b8f8) at mod_deflate.c:961
>>rv = -512
>>r = 0x7f8d78471750
>>ctx = 0x7f8d7864b9a8
>>len = 8096
>>blen = 1157663
>>data = 0x7f8d7e40b000 "/*! jQuery v2.1.4 | (c) 2005, 2015 jQuery
>> Foundation, Inc. | jquery.org/license
>> */\n!function(a,b){\"object\"==typeof module&&\"object\"==typeof
>> module.exports?module.exports=a.document?b(a,!0):function(a)"...
>>c = 0x55e62c7a7498
>> #13 0x55e62c044b1d in filter_harness (f=0x7f8d7846bc28, bb=0x80) at
>> mod_filter.c:323
>>ret = -512
>>cachecontrol = 0xfe00 > at address 0xfe00>
>>filter = 0xe
>> #14 0x55e62c002242 in default_handler (r=0x7f8d78471750) at core.c:4746
>>c = 0x7f8d7846b720
>>bb = 0x7f8d7864b8f8
>>e = 0xfe00
>>d = 0x7f8d7864dfa0
>>fd = 0x7f8d7864b7b8
>>bld_content_md5 = 2019866872
>> #15 0x55e62c014130 in ap_run_handler (r=r@entry=0x7f8d78471750) at
>> config.c:170
>>pHook = 
>>n = 11
>>rv = -1
>

Re: hanging apache httpd processes

[Stefan Priebe - Profihost AG]

in pthread_join () from
/lib/x86_64-linux-gnu/libpthread.so.0
No symbol table info available.
#1  0x7f8e80d9e4ab in apr_thread_join (retval=0x7ffda735e794,
thd=0x7f8de00dbb48) at threadproc/unix/thread.c:217
stat = 
thread_stat = 0x1
#2  0x55e62bfedeb3 in join_workers (listener=0x7f8de00dbb48,
threads=0x55e62cb2f960) at event.c:2376
thread_rv = 21990
#3  0x55e62bfee25a in child_main (child_num_arg=5,
child_bucket=749926752) at event.c:2574
threads = 0x55e62cb2f960
ts = 0x55e62c844430
thread_attr = 0x55e62c844450
start_thread_id = 0x55e62c8444b0
#4  0x55e62c0b8fe0 in make_child (s=0x7f8d7ef3d9d0, slot=5,
bucket=0) at event.c:2646
No locals.
#5  0x55e62c0b950c in server_main_loop (num_buckets=,
remaining_children_to_start=) at event.c:2916
ps = 0xfe00
status = 0
pid = {pid = 21894, in = 0x7f8e80d89d76 <find_entry+134>, out =
0x0, err = 0xc}
child_slot = 5
exitwhy = APR_PROC_EXIT
processed_status = 0
#6  event_run (_pconf=0x7f8d7ef3d9d0, plog=0x5, s=0xc8) at event.c:3035
remaining_children_to_start = 0
#7  0x55e62bff642e in ap_run_mpm (pconf=0x55e62c74f138,
plog=0x55e62c790bb8, s=0x55e62c7805a8) at mpm_common.c:94
pHook = 
n = 0
rv = -1
#8  0x55e62bfeed15 in main (argc=2, argv=0x7ffda735eb98) at main.c:783
c = 68 'D'
error = 0xfe00 
process = 0x55e62c74d218
pconf = 0x55e62c74f138
plog = 0x55e62c790bb8
ptemp = 0x55e62c7a8d98
pcommands = 0x55e62c771248
opt = 0x55e62c771338
mod = 0x55e62c3235d0 <ap_prelinked_modules+336>
opt_arg = 0x7ffda735eef1 "FOREGROUND"
signal_server = 0xfe00

Greets,
Stefan


Am 18.04.2017 um 14:47 schrieb Eric Covener:
> On Tue, Apr 18, 2017 at 8:43 AM, Stefan Priebe - Profihost AG
> <s.pri...@profihost.ag> wrote:
>> bt of such a process shows:
>> (gdb) bt
>> #0  0x7f5df74f64db in pthread_join () from
>> /lib/x86_64-linux-gnu/libpthread.so.0
> 
> Need to see the other threads in the process, this one is just waiting
> for the others to complete.
> 
>  "thread apply all bt full"
> 
> 

hanging apache httpd processes

[Stefan Priebe - Profihost AG]

Hi,

not sure whether this is related to mod_http2 v1.10.0 or is something else.

I've seen two servers where old httpd processes get stuck.

server-status looks like this:
SlotPID StoppingConnections Threads Async connections
total   accepting   busyidlewriting keep-alive  closing
0   27399   yes (old gen)   3   no  0   0   0   0   0
1   879 yes (old gen)   1   no  0   0   0   0   0
2   11609   yes (old gen)   2   no  0   0   0   0   0
3   9142yes (old gen)   2   no  0   0   0   0   0
4   11098   yes (old gen)   3   no  0   0   0   0   0
5   27400   yes (old gen)   1   no  0   0   0   0   0
6   16445   yes (old gen)   1   no  0   0   0   0   0
7   16446   yes (old gen)   1   no  0   0   0   0   0
8   7388yes (old gen)   4   no  0   0   0   0   0
9   24072   yes (old gen)   2   no  0   0   0   0   0
10  28673   yes (old gen)   2   no  0   0   0   0   0
11  7389yes (old gen)   3   no  0   0   0   0   0
12  11075   yes (old gen)   8   no  0   0   0   0   0
13  11076   yes (old gen)   6   no  0   0   0   0   0
14  26613   yes (old gen)   5   no  0   0   0   0   0
15  737 no  1   yes 8   192 0   0   1
16  739 no  3   yes 13  187 0   0   1
17  24059   yes (old gen)   1   no  0   0   0   0   0
18  26002   yes (old gen)   7   no  0   0   0   0   0
19  26003   yes (old gen)   4   no  0   0   0   0   0
20  2473yes (old gen)   2   no  0   0   0   0   0
21  3010yes (old gen)   4   no  0   0   0   0   0
22  8847yes (old gen)   3   no  0   0   0   0   0
Sum 23  21  69  21  379 0   0   2

bt of such a process shows:
(gdb) bt
#0  0x7f5df74f64db in pthread_join () from
/lib/x86_64-linux-gnu/libpthread.so.0
#1  0x7f5df77314ab in apr_thread_join (retval=0x7ffc0878f424,
thd=0x7f5d580dc7c8) at threadproc/unix/thread.c:217
#2  0x555c1dc9deb3 in join_workers (listener=0x7f5d580dc7c8,
threads=0x555c1f3d4310) at event.c:2376
#3  0x555c1dc9e25a in child_main (child_num_arg=0,
child_bucket=524108560) at event.c:2574
#4  0x555c1dd68fe0 in make_child (s=0x7f5cf673c9d0, slot=0,
bucket=0) at event.c:2646
#5  0x555c1dd6950c in server_main_loop (num_buckets=,
remaining_children_to_start=) at event.c:2916
#6  event_run (_pconf=0x7f5cf673c9d0, plog=0x0, s=0xc8) at event.c:3035
#7  0x555c1dca642e in ap_run_mpm (pconf=0x555c1f031138,
plog=0x555c1f072bb8, s=0x555c1f0625a8) at mpm_common.c:94
#8  0x555c1dc9ed15 in main (argc=2, argv=0x7ffc0878f828) at main.c:783


An strace is like this one:
strace -ff -p 32375
Process 32375 attached with 303 threads
[pid   856] futex(0x7f5ce8003c60, FUTEX_WAIT_PRIVATE, 2, NULL

[pid   486] futex(0x555c1f3b17d4, FUTEX_WAIT_PRIVATE, 1, NULL

[pid   857] epoll_wait(14,  
[pid   484] futex(0x555c1f3b174c, FUTEX_WAIT_PRIVATE, 1, NULL

[pid   482] futex(0x555c1f3b16c4, FUTEX_WAIT_PRIVATE, 1, NULL

[pid   481] futex(0x555c1f3b163c, FUTEX_WAIT_PRIVATE, 1, NULL

...


Greets,
Stefan

Re: Apache 2.4.26 AH: scoreboard is full, not at MaxRequestWorkers.Increase,ServerLimit

[Stefan Priebe - Profihost AG]

Hi,

Am 16.03.2017 um 18:05 schrieb Luca Toscano:
> 
> 
> 2017-03-16 15:24 GMT+01:00 Stefan Priebe - Profihost AG
> <s.pri...@profihost.ag <mailto:s.pri...@profihost.ag>>:
> 
> Hi,
> 
> Am 16.03.2017 um 12:26 schrieb Luca Toscano:
> > Hi Stefan,
>     >
> > 2017-03-16 12:14 GMT+01:00 Stefan Priebe - Profihost AG
> > <s.pri...@profihost.ag <mailto:s.pri...@profihost.ag>
> <mailto:s.pri...@profihost.ag <mailto:s.pri...@profihost.ag>>>:
> >
> > Hi Yann,
> >
> > no sure whether this is due to your mpm event patch.
> >
> > From time to time i see the following error mesages in my logs
> and the
> > only chance to fix it is to restart apache.
> >
> > [Thu Mar 16 01:00:35.445184 2017] [mpm_event:error] [pid 27485:tid
> > 140212799559552] AH: scoreboard is full, not at
> > MaxRequestWorkers.Increase
> > ServerLimit.
> > [Thu Mar 16 01:00:36.446178 2017] [mpm_event:error] [pid 27485:tid
> > 140212799559552] AH: scoreboard is full, not at
> > MaxRequestWorkers.Increase
> > ServerLimit.
> > [Thu Mar 16 01:00:37.447181 2017] [mpm_event:error] [pid 27485:tid
> > 140212799559552] AH: scoreboard is full, not at
> > MaxRequestWorkers.Increase
> > ServerLimit.
> > [Thu Mar 16 01:00:38.448177 2017] [mpm_event:error] [pid 27485:tid
> > 140212799559552] AH: scoreboard is full, not at
> > MaxRequestWorkers.Increase
> > ServerLimit.
> > [Thu Mar 16 01:00:39.449185 2017] [mpm_event:error] [pid 27485:tid
> > 140212799559552] AH: scoreboard is full, not at
> > MaxRequestWorkers.Increase
> > ServerLimit.
> > [Thu Mar 16 01:00:40.450184 2017] [mpm_event:error] [pid 27485:tid
> > 140212799559552] AH: scoreboard is full, not at
> > MaxRequestWorkers.Increase
> > ServerLimit.
> >
> > Settings:
> > ServerLimit   50
> > ThreadLimit   200
> > ThreadsPerChild200
> > MinSpareThreads200
> > MaxSpareThreads400
> > MaxClients 1
> > MaxRequestsPerChild1
> >
> > MaxRequestWorkers isn't set at all.
> >
> >
> > I believe that MaxClients (its old name) takes the place of
> > MaxRequestWorkers, but it is set as the default value (ServerLimit *
> > ThreadsPerChild). From 2.4.25 onwards mpm-event offers a new
> > functionality to handle Gracefully terminating processes
> > (https://httpd.apache.org/docs/current/mod/event.html#how-it-works
> <https://httpd.apache.org/docs/current/mod/event.html#how-it-works>), but
> > it must be tuned raising the value of ServerLimit (more details in the
> > docs).
> 
> Hui didn't know this. So i should remove ALL settings and only set:
> ThreadsPerChild
> ServerLimit
> MaxRequestWorkers
> AsyncRequestWorkerFactor
> 
> ? is this true? I'm missing some examples also considering MinSpare and
> ThreadLimit or are they no longer needed?
> 
> 
> I would simply replace MaxClients with MaxRequestWorkers keeping the
> rest of your config (that it is still valid, all the settings that you
> mentioned are still used by mpm-event), without touching
>  AsyncRequestWorkerFactor (unless you want to play with it but the
> default is generally good). About your specific max scoreboard issue, I
> would:
> 
> 1) try to raise ServerLimit to allow more space for slots occupied by
> processes in G state (graceful termination), as indicated in the docs.
> 2) Follow Daniel's suggestion about Max/Min spare workers.

Will try to fix this. Need to implement this in our env. I'll most
probably need until Monday.


> > This is only a speculation from my side, to have a better idea of what's
> > happening it would be great to see how the Scoreboard looks like in
> > server status, and if the error status happens during specific events
> > like graceful reload for log rotation.
> 
> All of them happened after a reload - but I'll recheck. I'm pretty sure
> that the /server-status page was no longer responding. Is there any
> other way to get the status of of httpd while it does no longer
> serve pages?
> 
> 
> I am not aware of any other way sadly, but hopefully you will not need
> it with the new settings :)
> 
> Do you have long running http connections that can keep httpd processes
> in the G state after reload? Usually this is the main problem, and very
> easy to test.
> 
> Let us know how it goes!
> 
> Luca 
> 

Re: Apache 2.4.26 AH: scoreboard is full, not at MaxRequestWorkers.Increase,ServerLimit

[Stefan Priebe - Profihost AG]

Hi,

Am 16.03.2017 um 12:26 schrieb Luca Toscano:
> Hi Stefan,
> 
> 2017-03-16 12:14 GMT+01:00 Stefan Priebe - Profihost AG
> <s.pri...@profihost.ag <mailto:s.pri...@profihost.ag>>:
> 
> Hi Yann,
> 
> no sure whether this is due to your mpm event patch.
> 
> From time to time i see the following error mesages in my logs and the
> only chance to fix it is to restart apache.
> 
> [Thu Mar 16 01:00:35.445184 2017] [mpm_event:error] [pid 27485:tid
> 140212799559552] AH: scoreboard is full, not at
> MaxRequestWorkers.Increase
> ServerLimit.
> [Thu Mar 16 01:00:36.446178 2017] [mpm_event:error] [pid 27485:tid
> 140212799559552] AH: scoreboard is full, not at
> MaxRequestWorkers.Increase
> ServerLimit.
> [Thu Mar 16 01:00:37.447181 2017] [mpm_event:error] [pid 27485:tid
> 140212799559552] AH: scoreboard is full, not at
> MaxRequestWorkers.Increase
> ServerLimit.
> [Thu Mar 16 01:00:38.448177 2017] [mpm_event:error] [pid 27485:tid
> 140212799559552] AH: scoreboard is full, not at
> MaxRequestWorkers.Increase
> ServerLimit.
> [Thu Mar 16 01:00:39.449185 2017] [mpm_event:error] [pid 27485:tid
> 140212799559552] AH: scoreboard is full, not at
> MaxRequestWorkers.Increase
> ServerLimit.
> [Thu Mar 16 01:00:40.450184 2017] [mpm_event:error] [pid 27485:tid
> 140212799559552] AH: scoreboard is full, not at
> MaxRequestWorkers.Increase
> ServerLimit.
> 
> Settings:
> ServerLimit   50
> ThreadLimit   200
> ThreadsPerChild200
> MinSpareThreads200
> MaxSpareThreads400
> MaxClients 1
> MaxRequestsPerChild1
> 
> MaxRequestWorkers isn't set at all.
> 
> 
> I believe that MaxClients (its old name) takes the place of
> MaxRequestWorkers, but it is set as the default value (ServerLimit *
> ThreadsPerChild). From 2.4.25 onwards mpm-event offers a new
> functionality to handle Gracefully terminating processes
> (https://httpd.apache.org/docs/current/mod/event.html#how-it-works), but
> it must be tuned raising the value of ServerLimit (more details in the
> docs).

Hui didn't know this. So i should remove ALL settings and only set:
ThreadsPerChild
ServerLimit
MaxRequestWorkers
AsyncRequestWorkerFactor

? is this true? I'm missing some examples also considering MinSpare and
ThreadLimit or are they no longer needed?

> This is only a speculation from my side, to have a better idea of what's
> happening it would be great to see how the Scoreboard looks like in
> server status, and if the error status happens during specific events
> like graceful reload for log rotation. 

All of them happened after a reload - but I'll recheck. I'm pretty sure
that the /server-status page was no longer responding. Is there any
other way to get the status of of httpd while it does no longer serve pages?

> Hope that helps! As always, thanks a lot for your tests :)

Greets,
Stefan

> Luca 

Apache 2.4.26 AH: scoreboard is full, not at MaxRequestWorkers.Increase,ServerLimit

[Stefan Priebe - Profihost AG]

Hi Yann,

no sure whether this is due to your mpm event patch.

>From time to time i see the following error mesages in my logs and the
only chance to fix it is to restart apache.

[Thu Mar 16 01:00:35.445184 2017] [mpm_event:error] [pid 27485:tid
140212799559552] AH: scoreboard is full, not at MaxRequestWorkers.Increase
ServerLimit.
[Thu Mar 16 01:00:36.446178 2017] [mpm_event:error] [pid 27485:tid
140212799559552] AH: scoreboard is full, not at MaxRequestWorkers.Increase
ServerLimit.
[Thu Mar 16 01:00:37.447181 2017] [mpm_event:error] [pid 27485:tid
140212799559552] AH: scoreboard is full, not at MaxRequestWorkers.Increase
ServerLimit.
[Thu Mar 16 01:00:38.448177 2017] [mpm_event:error] [pid 27485:tid
140212799559552] AH: scoreboard is full, not at MaxRequestWorkers.Increase
ServerLimit.
[Thu Mar 16 01:00:39.449185 2017] [mpm_event:error] [pid 27485:tid
140212799559552] AH: scoreboard is full, not at MaxRequestWorkers.Increase
ServerLimit.
[Thu Mar 16 01:00:40.450184 2017] [mpm_event:error] [pid 27485:tid
140212799559552] AH: scoreboard is full, not at MaxRequestWorkers.Increase
ServerLimit.

Settings:
ServerLimit   50
ThreadLimit   200
ThreadsPerChild200
MinSpareThreads200
MaxSpareThreads400
MaxClients 1
MaxRequestsPerChild1

MaxRequestWorkers isn't set at all.

Greets,
Stefan

Re: mod_http2 v1.9.3

[Stefan Priebe - Profihost AG]

06a60)
at http_request.c:436
#17 0x55dcf0c68d90 in ap_process_request (r=0x7f856a60)
at http_request.c:471
#18 0x55dcf0ca9dc2 in h2_task_process_request (c=,
task=) at h2_task.c:622
#19 h2_task_process_conn (c=0x7f8569e8) at h2_task.c:669
#20 0x55dcf0c39070 in ap_run_process_connection (c=0x7f850a30)
at connection.c:42
#21 0x55dcf0cab29f in h2_task_do (task=0x7f854a50,
thread=0x55dcf235ce60, worker_id=0) at h2_task.c:579
#22 0x55dcf0cb4c52 in execute (thread=0x55dcf235ce60,
wctx=0x55dcf235ce20)
at h2_worker.c:46
#23 0x7f85d4bcd0a4 in start_thread ()
   from /lib/x86_64-linux-gnu/libpthread.so.0
#24 0x7f85d490262d in clone () from /lib/x86_64-linux-gnu/libc.so.6

Stefan
Am 12.03.2017 um 19:53 schrieb Stefan Priebe - Profihost AG:
> Thanks. Installed.
> 
> Greets,
> Stefan
> Am 12.03.2017 um 15:58 schrieb Stefan Eissing:
>> Stefan,
>>
>> I would be very interested in hearing how v1.9.3 fares in your environment:
>> - I found and fixed a race in connection shutdown that could potentially 
>> explain your sometimes observed crashes
>> - I changed mutex use for individual requests, producing less contention. 
>> Tests on my system are fine, need to see some real world exposure (read: I 
>> warned you)
>>
>> Available at: https://github.com/icing/mod_h2/releases/tag/v1.9.3
>>
>> Thanks a lot!
>>
>> Stefan Eissing
>>
>> bytes GmbH
>> Hafenstrasse 16
>> 48155 Münster
>> www.greenbytes.de
>>

Re: mod_http2 v1.9.3

[Stefan Priebe - Profihost AG]

Thanks. Installed.

Greets,
Stefan
Am 12.03.2017 um 15:58 schrieb Stefan Eissing:
> Stefan,
> 
> I would be very interested in hearing how v1.9.3 fares in your environment:
> - I found and fixed a race in connection shutdown that could potentially 
> explain your sometimes observed crashes
> - I changed mutex use for individual requests, producing less contention. 
> Tests on my system are fine, need to see some real world exposure (read: I 
> warned you)
> 
> Available at: https://github.com/icing/mod_h2/releases/tag/v1.9.3
> 
> Thanks a lot!
> 
> Stefan Eissing
> 
> bytes GmbH
> Hafenstrasse 16
> 48155 Münster
> www.greenbytes.de
> 

Re: release v1.9.2

[Stefan Priebe - Profihost AG]

Am 09.03.2017 um 18:09 schrieb Stefan Eissing:
> Thanks for running our patches with many changes! Not a mistake to have it 
> running fine for four days. ;-)
> 
> Hmm, so we hunt a rare beast. After much effort from all sides we made it 
> rare. So, nothing to be ashamed of. Need to keep looking...

May be Yann has an idea? May be it's not related to http2 at all?

Stefan

>> Am 09.03.2017 um 17:32 schrieb Stefan Priebe - Profihost AG 
>> <s.pri...@profihost.ag>:
>>
>> Hi Stefan,
>>
>> while doing longer testing i can say that also the version which was
>> working for 4 days segfaults. So it was never solved ;-( Sorry about
>> that. Testing was just too short.
>>
>> Greets,
>> Stefan
>>
>> Am 05.03.2017 um 14:49 schrieb Stefan Priebe - Profihost AG:
>>> Hi Stefan,
>>>
>>> yes this seems to correct BUT i'm not sure i the test was long enough
>>> where i hadn't a segfault. I'll rerun a test with that version. To
>>> verify this was no just a "fault".
>>>
>>> Greets,
>>> Stefan
>>> Am 05.03.2017 um 14:34 schrieb Stefan Eissing:
>>>> Thanks. I try to summarize:
>>>>
>>>> - we did see this rarely with unpatched v1.9.2 and Yann's mpm patch v??
>>>> - we still see this rarely with the patch adding mutex to the main conn 
>>>> allocator (so this seems not to change anything)
>>>> - we did not see this with v1.9.2 and Yann's patch on ptrans version  ??
>>>>
>>>> Is this correct?
>>>>
>>>>> Am 04.03.2017 um 22:34 schrieb Stefan Priebe - Profihost AG 
>>>>> <s.pri...@profihost.ag>:
>>>>>
>>>>> Hi Stefan,
>>>>>
>>>>> current trace - not sure whether this is http2 related at all:
>>>>>
>>>>> Core was generated by `/usr/local/apache/bin/httpd -DFOREGROUND'.
>>>>> Program terminated with signal SIGSEGV, Segmentation fault.
>>>>> #0  allocator_free (node=0x0, allocator=0x7fbf4c05a410)
>>>>>   at memory/unix/apr_pools.c:381
>>>>> #0  allocator_free (node=0x0, allocator=0x7fbf4c05a410)
>>>>>   at memory/unix/apr_pools.c:381
>>>>> #1  apr_pool_clear (pool=0x7fbf4c04f888) at memory/unix/apr_pools.c:793
>>>>> #2  0x5642878c2818 in ap_push_pool (queue_info=0x0,
>>>>>   pool_to_recycle=0x7fbf4c05a418) at fdqueue.c:234
>>>>> #3  0x5642878bdb5a in process_lingering_close (cs=0x7fbf4c04fb18,
>>>>>   pfd=0x5642886c7078) at event.c:1513
>>>>> #4  0x5642878c1690 in listener_thread (thd=0x0, dummy=0x549eb52311a6f)
>>>>>   at event.c:1837
>>>>> #5  0x00007fbf5f2940a4 in start_thread ()
>>>>>  from /lib/x86_64-linux-gnu/libpthread.so.0
>>>>> #6  0x7fbf5efc962d in clone () from /lib/x86_64-linux-gnu/libc.so.6
>>>>>
>>>>> Stefan
>>>>>
>>>>>> Am 03.03.2017 um 21:02 schrieb Stefan Priebe - Profihost AG:
>>>>>> Hi Stefan,
>>>>>>
>>>>>> live. Sorry for the late reply.
>>>>>>
>>>>>> Stefan
>>>>>>
>>>>>>> Am 28.02.2017 um 11:49 schrieb Stefan Eissing:
>>>>>>> Meh, my mistake. Sorry about that. Did not cleanup properly. Dare you 
>>>>>>> with an improved version:
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>> Am 28.02.2017 um 08:41 schrieb Stefan Priebe - Profihost AG 
>>>>>>>> <s.pri...@profihost.ag>:
>>>>>>>>
>>>>>>>> That one breaks everyting. Multiple crashes per second.
>>>>>>>>
>>>>>>>> [Thread debugging using libthread_db enabled]
>>>>>>>> Using host libthread_db library 
>>>>>>>> "/lib/x86_64-linux-gnu/libthread_db.so.1".
>>>>>>>> Core was generated by `/usr/local/apache/bin/httpd -DFOREGROUND'.
>>>>>>>> Program terminated with signal SIGABRT, Aborted.
>>>>>>>> #0  0x7f02724ea067 in raise () from /lib/x86_64-linux-gnu/libc.so.6
>>>>>>>> #0  0x7f02724ea067 in raise () from /lib/x86_64-linux-gnu/libc.so.6
>>>>>>>> #1  0x7f02724eb448 in abort () from /lib/x86_64-linux-gnu/libc.so.6
>>>>>>>>

Re: release v1.9.2

[Stefan Priebe - Profihost AG]

Hi Stefan,

while doing longer testing i can say that also the version which was
working for 4 days segfaults. So it was never solved ;-( Sorry about
that. Testing was just too short.

Greets,
Stefan

Am 05.03.2017 um 14:49 schrieb Stefan Priebe - Profihost AG:
> Hi Stefan,
> 
> yes this seems to correct BUT i'm not sure i the test was long enough
> where i hadn't a segfault. I'll rerun a test with that version. To
> verify this was no just a "fault".
> 
> Greets,
> Stefan
> Am 05.03.2017 um 14:34 schrieb Stefan Eissing:
>> Thanks. I try to summarize:
>>
>> - we did see this rarely with unpatched v1.9.2 and Yann's mpm patch v??
>> - we still see this rarely with the patch adding mutex to the main conn 
>> allocator (so this seems not to change anything)
>> - we did not see this with v1.9.2 and Yann's patch on ptrans version  ??
>>
>> Is this correct?
>>
>>> Am 04.03.2017 um 22:34 schrieb Stefan Priebe - Profihost AG 
>>> <s.pri...@profihost.ag>:
>>>
>>> Hi Stefan,
>>>
>>> current trace - not sure whether this is http2 related at all:
>>>
>>> Core was generated by `/usr/local/apache/bin/httpd -DFOREGROUND'.
>>> Program terminated with signal SIGSEGV, Segmentation fault.
>>> #0  allocator_free (node=0x0, allocator=0x7fbf4c05a410)
>>>at memory/unix/apr_pools.c:381
>>> #0  allocator_free (node=0x0, allocator=0x7fbf4c05a410)
>>>at memory/unix/apr_pools.c:381
>>> #1  apr_pool_clear (pool=0x7fbf4c04f888) at memory/unix/apr_pools.c:793
>>> #2  0x5642878c2818 in ap_push_pool (queue_info=0x0,
>>>pool_to_recycle=0x7fbf4c05a418) at fdqueue.c:234
>>> #3  0x5642878bdb5a in process_lingering_close (cs=0x7fbf4c04fb18,
>>>pfd=0x5642886c7078) at event.c:1513
>>> #4  0x5642878c1690 in listener_thread (thd=0x0, dummy=0x549eb52311a6f)
>>>at event.c:1837
>>> #5  0x7fbf5f2940a4 in start_thread ()
>>>   from /lib/x86_64-linux-gnu/libpthread.so.0
>>> #6  0x7fbf5efc962d in clone () from /lib/x86_64-linux-gnu/libc.so.6
>>>
>>> Stefan
>>>
>>>> Am 03.03.2017 um 21:02 schrieb Stefan Priebe - Profihost AG:
>>>> Hi Stefan,
>>>>
>>>> live. Sorry for the late reply.
>>>>
>>>> Stefan
>>>>
>>>>> Am 28.02.2017 um 11:49 schrieb Stefan Eissing:
>>>>> Meh, my mistake. Sorry about that. Did not cleanup properly. Dare you 
>>>>> with an improved version:
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>> Am 28.02.2017 um 08:41 schrieb Stefan Priebe - Profihost AG 
>>>>>> <s.pri...@profihost.ag>:
>>>>>>
>>>>>> That one breaks everyting. Multiple crashes per second.
>>>>>>
>>>>>> [Thread debugging using libthread_db enabled]
>>>>>> Using host libthread_db library 
>>>>>> "/lib/x86_64-linux-gnu/libthread_db.so.1".
>>>>>> Core was generated by `/usr/local/apache/bin/httpd -DFOREGROUND'.
>>>>>> Program terminated with signal SIGABRT, Aborted.
>>>>>> #0  0x7f02724ea067 in raise () from /lib/x86_64-linux-gnu/libc.so.6
>>>>>> #0  0x7f02724ea067 in raise () from /lib/x86_64-linux-gnu/libc.so.6
>>>>>> #1  0x7f02724eb448 in abort () from /lib/x86_64-linux-gnu/libc.so.6
>>>>>> #2  0x7f02724e3266 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
>>>>>> #3  0x7f02724e3312 in __assert_fail () from
>>>>>> /lib/x86_64-linux-gnu/libc.so.6
>>>>>> #4  0x7f027287062f in __pthread_tpp_change_priority ()
>>>>>>  from /lib/x86_64-linux-gnu/libpthread.so.0
>>>>>> #5  0x7f0272865e9f in __pthread_mutex_lock_full ()
>>>>>>  from /lib/x86_64-linux-gnu/libpthread.so.0
>>>>>> #6  0x7f0272a98db8 in allocator_alloc (in_size=in_size@entry=8032,
>>>>>>   allocator=0x7f025c03f3c0) at memory/unix/apr_pools.c:244
>>>>>> #7  apr_allocator_alloc (allocator=0x7f025c03f3c0, size=size@entry=8032)
>>>>>>   at memory/unix/apr_pools.c:438
>>>>>> #8  0x7f0272cbcac4 in apr_bucket_alloc (size=8032, size@entry=8000,
>>>>>>   list=0x7f022c0008e8) at buckets/apr_buckets_alloc.c:157
>>>>>> #9  0x7f0272cbdca2 in socket_bucket_read (a=0x7f022c000b08,
>>>>>>   str=0x7f02627a87b8, len=0x7f02627a87c0, blo

Re: release v1.9.2

[Stefan Priebe - Profihost AG]

Hi Stefan,

yes this seems to correct BUT i'm not sure i the test was long enough
where i hadn't a segfault. I'll rerun a test with that version. To
verify this was no just a "fault".

Greets,
Stefan
Am 05.03.2017 um 14:34 schrieb Stefan Eissing:
> Thanks. I try to summarize:
> 
> - we did see this rarely with unpatched v1.9.2 and Yann's mpm patch v??
> - we still see this rarely with the patch adding mutex to the main conn 
> allocator (so this seems not to change anything)
> - we did not see this with v1.9.2 and Yann's patch on ptrans version  ??
> 
> Is this correct?
> 
>> Am 04.03.2017 um 22:34 schrieb Stefan Priebe - Profihost AG 
>> <s.pri...@profihost.ag>:
>>
>> Hi Stefan,
>>
>> current trace - not sure whether this is http2 related at all:
>>
>> Core was generated by `/usr/local/apache/bin/httpd -DFOREGROUND'.
>> Program terminated with signal SIGSEGV, Segmentation fault.
>> #0  allocator_free (node=0x0, allocator=0x7fbf4c05a410)
>>at memory/unix/apr_pools.c:381
>> #0  allocator_free (node=0x0, allocator=0x7fbf4c05a410)
>>at memory/unix/apr_pools.c:381
>> #1  apr_pool_clear (pool=0x7fbf4c04f888) at memory/unix/apr_pools.c:793
>> #2  0x5642878c2818 in ap_push_pool (queue_info=0x0,
>>pool_to_recycle=0x7fbf4c05a418) at fdqueue.c:234
>> #3  0x5642878bdb5a in process_lingering_close (cs=0x7fbf4c04fb18,
>>pfd=0x5642886c7078) at event.c:1513
>> #4  0x5642878c1690 in listener_thread (thd=0x0, dummy=0x549eb52311a6f)
>>at event.c:1837
>> #5  0x7fbf5f2940a4 in start_thread ()
>>   from /lib/x86_64-linux-gnu/libpthread.so.0
>> #6  0x7fbf5efc962d in clone () from /lib/x86_64-linux-gnu/libc.so.6
>>
>> Stefan
>>
>>> Am 03.03.2017 um 21:02 schrieb Stefan Priebe - Profihost AG:
>>> Hi Stefan,
>>>
>>> live. Sorry for the late reply.
>>>
>>> Stefan
>>>
>>>> Am 28.02.2017 um 11:49 schrieb Stefan Eissing:
>>>> Meh, my mistake. Sorry about that. Did not cleanup properly. Dare you with 
>>>> an improved version:
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>> Am 28.02.2017 um 08:41 schrieb Stefan Priebe - Profihost AG 
>>>>> <s.pri...@profihost.ag>:
>>>>>
>>>>> That one breaks everyting. Multiple crashes per second.
>>>>>
>>>>> [Thread debugging using libthread_db enabled]
>>>>> Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
>>>>> Core was generated by `/usr/local/apache/bin/httpd -DFOREGROUND'.
>>>>> Program terminated with signal SIGABRT, Aborted.
>>>>> #0  0x7f02724ea067 in raise () from /lib/x86_64-linux-gnu/libc.so.6
>>>>> #0  0x7f02724ea067 in raise () from /lib/x86_64-linux-gnu/libc.so.6
>>>>> #1  0x7f02724eb448 in abort () from /lib/x86_64-linux-gnu/libc.so.6
>>>>> #2  0x7f02724e3266 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
>>>>> #3  0x7f02724e3312 in __assert_fail () from
>>>>> /lib/x86_64-linux-gnu/libc.so.6
>>>>> #4  0x7f027287062f in __pthread_tpp_change_priority ()
>>>>>  from /lib/x86_64-linux-gnu/libpthread.so.0
>>>>> #5  0x7f0272865e9f in __pthread_mutex_lock_full ()
>>>>>  from /lib/x86_64-linux-gnu/libpthread.so.0
>>>>> #6  0x7f0272a98db8 in allocator_alloc (in_size=in_size@entry=8032,
>>>>>   allocator=0x7f025c03f3c0) at memory/unix/apr_pools.c:244
>>>>> #7  apr_allocator_alloc (allocator=0x7f025c03f3c0, size=size@entry=8032)
>>>>>   at memory/unix/apr_pools.c:438
>>>>> #8  0x7f0272cbcac4 in apr_bucket_alloc (size=8032, size@entry=8000,
>>>>>   list=0x7f022c0008e8) at buckets/apr_buckets_alloc.c:157
>>>>> #9  0x7f0272cbdca2 in socket_bucket_read (a=0x7f022c000b08,
>>>>>   str=0x7f02627a87b8, len=0x7f02627a87c0, block=)
>>>>>   at buckets/apr_buckets_socket.c:34
>>>>> #10 0x565098cf1fb1 in ap_core_input_filter (f=0x7f022c0056b0,
>>>>>   b=0x7f022c005630, mode=, block=APR_NONBLOCK_READ,
>>>>>   readbytes=5) at core_filters.c:235
>>>>> #11 0x565098d3aaca in logio_in_filter (f=,
>>>>>   bb=0x7f022c005630, mode=, block=,
>>>>>   readbytes=) at mod_logio.c:165
>>>>> #12 0x565098d45b9a in bio_filter_in_read (bio=0x7f024800b460,
>>>>>   in=0x7f02480492a3 "", inlen=5) at ssl_engine_io.c:483

Re: release v1.9.2

[Stefan Priebe - Profihost AG]

Hi Stefan,

current trace - not sure whether this is http2 related at all:

Core was generated by `/usr/local/apache/bin/httpd -DFOREGROUND'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  allocator_free (node=0x0, allocator=0x7fbf4c05a410)
at memory/unix/apr_pools.c:381
#0  allocator_free (node=0x0, allocator=0x7fbf4c05a410)
at memory/unix/apr_pools.c:381
#1  apr_pool_clear (pool=0x7fbf4c04f888) at memory/unix/apr_pools.c:793
#2  0x5642878c2818 in ap_push_pool (queue_info=0x0,
pool_to_recycle=0x7fbf4c05a418) at fdqueue.c:234
#3  0x5642878bdb5a in process_lingering_close (cs=0x7fbf4c04fb18,
pfd=0x5642886c7078) at event.c:1513
#4  0x5642878c1690 in listener_thread (thd=0x0, dummy=0x549eb52311a6f)
at event.c:1837
#5  0x7fbf5f2940a4 in start_thread ()
   from /lib/x86_64-linux-gnu/libpthread.so.0
#6  0x7fbf5efc962d in clone () from /lib/x86_64-linux-gnu/libc.so.6

Stefan

Am 03.03.2017 um 21:02 schrieb Stefan Priebe - Profihost AG:
> Hi Stefan,
> 
> live. Sorry for the late reply.
> 
> Stefan
> 
> Am 28.02.2017 um 11:49 schrieb Stefan Eissing:
>> Meh, my mistake. Sorry about that. Did not cleanup properly. Dare you with 
>> an improved version:
>>
>>
>>
>>
>>
>>> Am 28.02.2017 um 08:41 schrieb Stefan Priebe - Profihost AG 
>>> <s.pri...@profihost.ag>:
>>>
>>> That one breaks everyting. Multiple crashes per second.
>>>
>>> [Thread debugging using libthread_db enabled]
>>> Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
>>> Core was generated by `/usr/local/apache/bin/httpd -DFOREGROUND'.
>>> Program terminated with signal SIGABRT, Aborted.
>>> #0  0x7f02724ea067 in raise () from /lib/x86_64-linux-gnu/libc.so.6
>>> #0  0x7f02724ea067 in raise () from /lib/x86_64-linux-gnu/libc.so.6
>>> #1  0x7f02724eb448 in abort () from /lib/x86_64-linux-gnu/libc.so.6
>>> #2  0x7f02724e3266 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
>>> #3  0x7f02724e3312 in __assert_fail () from
>>> /lib/x86_64-linux-gnu/libc.so.6
>>> #4  0x7f027287062f in __pthread_tpp_change_priority ()
>>>   from /lib/x86_64-linux-gnu/libpthread.so.0
>>> #5  0x7f0272865e9f in __pthread_mutex_lock_full ()
>>>   from /lib/x86_64-linux-gnu/libpthread.so.0
>>> #6  0x7f0272a98db8 in allocator_alloc (in_size=in_size@entry=8032,
>>>allocator=0x7f025c03f3c0) at memory/unix/apr_pools.c:244
>>> #7  apr_allocator_alloc (allocator=0x7f025c03f3c0, size=size@entry=8032)
>>>at memory/unix/apr_pools.c:438
>>> #8  0x7f0272cbcac4 in apr_bucket_alloc (size=8032, size@entry=8000,
>>>list=0x7f022c0008e8) at buckets/apr_buckets_alloc.c:157
>>> #9  0x7f0272cbdca2 in socket_bucket_read (a=0x7f022c000b08,
>>>str=0x7f02627a87b8, len=0x7f02627a87c0, block=)
>>>at buckets/apr_buckets_socket.c:34
>>> #10 0x565098cf1fb1 in ap_core_input_filter (f=0x7f022c0056b0,
>>>b=0x7f022c005630, mode=, block=APR_NONBLOCK_READ,
>>>readbytes=5) at core_filters.c:235
>>> #11 0x565098d3aaca in logio_in_filter (f=,
>>>bb=0x7f022c005630, mode=, block=,
>>>readbytes=) at mod_logio.c:165
>>> #12 0x565098d45b9a in bio_filter_in_read (bio=0x7f024800b460,
>>>in=0x7f02480492a3 "", inlen=5) at ssl_engine_io.c:483
>>> #13 0x7f02736b014c in BIO_read ()
>>>   from /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.0
>>> #14 0x7f0273a13c92 in ?? () from
>>> /usr/lib/x86_64-linux-gnu/libssl.so.1.0.0
>>> #15 0x7f0273a1548d in ?? () from
>>> /usr/lib/x86_64-linux-gnu/libssl.so.1.0.0
>>> #16 0x7f0273a12024 in ?? () from
>>> /usr/lib/x86_64-linux-gnu/libssl.so.1.0.0
>>> #17 0x565098d469f3 in ssl_io_input_read (inctx=0x7f022c0035b8,
>>>buf=0x7f022c003600 "GET
>>> /media/image/07/3c/c2/4612_13721_160192280322.jpeg HTTP/1.1\r\nHost:
>>> www.onlinestore.it\r\nUser-Agent: curl/7.15+ (x64-criteo) libcurl/7.15+
>>> OpenSSL zlib libidn\r\nAccept: */*\r\n\r\n", len=0x7f02627a8b38)
>>>at ssl_engine_io.c:614
>>> #18 0x565098d493ec in ssl_io_filter_input (f=0x7f022c005608,
>>>bb=0x7f022c006dd0, mode=, block=,
>>>readbytes=8000) at ssl_engine_io.c:1474
>>> #19 0x565098d5cd85 in h2_filter_core_input (f=0x7f022c005980,
>>>brigade=0x7f022c006dd0, mode=738225624, block=APR_NONBLOCK_READ,
>>>readbytes=8000) at h2_filter.c:149
>>> #20 0x565098d6809b in h2_session_read (session=0x7f022c006920,
>>>block

Re: release v1.9.2

[Stefan Priebe - Profihost AG]

Hi Stefan,

live. Sorry for the late reply.

Stefan

Am 28.02.2017 um 11:49 schrieb Stefan Eissing:
> Meh, my mistake. Sorry about that. Did not cleanup properly. Dare you with an 
> improved version:
> 
> 
> 
> 
> 
>> Am 28.02.2017 um 08:41 schrieb Stefan Priebe - Profihost AG 
>> <s.pri...@profihost.ag>:
>>
>> That one breaks everyting. Multiple crashes per second.
>>
>> [Thread debugging using libthread_db enabled]
>> Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
>> Core was generated by `/usr/local/apache/bin/httpd -DFOREGROUND'.
>> Program terminated with signal SIGABRT, Aborted.
>> #0  0x7f02724ea067 in raise () from /lib/x86_64-linux-gnu/libc.so.6
>> #0  0x7f02724ea067 in raise () from /lib/x86_64-linux-gnu/libc.so.6
>> #1  0x7f02724eb448 in abort () from /lib/x86_64-linux-gnu/libc.so.6
>> #2  0x7f02724e3266 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
>> #3  0x7f02724e3312 in __assert_fail () from
>> /lib/x86_64-linux-gnu/libc.so.6
>> #4  0x7f027287062f in __pthread_tpp_change_priority ()
>>   from /lib/x86_64-linux-gnu/libpthread.so.0
>> #5  0x7f0272865e9f in __pthread_mutex_lock_full ()
>>   from /lib/x86_64-linux-gnu/libpthread.so.0
>> #6  0x7f0272a98db8 in allocator_alloc (in_size=in_size@entry=8032,
>>allocator=0x7f025c03f3c0) at memory/unix/apr_pools.c:244
>> #7  apr_allocator_alloc (allocator=0x7f025c03f3c0, size=size@entry=8032)
>>at memory/unix/apr_pools.c:438
>> #8  0x7f0272cbcac4 in apr_bucket_alloc (size=8032, size@entry=8000,
>>list=0x7f022c0008e8) at buckets/apr_buckets_alloc.c:157
>> #9  0x7f0272cbdca2 in socket_bucket_read (a=0x7f022c000b08,
>>str=0x7f02627a87b8, len=0x7f02627a87c0, block=)
>>at buckets/apr_buckets_socket.c:34
>> #10 0x565098cf1fb1 in ap_core_input_filter (f=0x7f022c0056b0,
>>b=0x7f022c005630, mode=, block=APR_NONBLOCK_READ,
>>readbytes=5) at core_filters.c:235
>> #11 0x565098d3aaca in logio_in_filter (f=,
>>bb=0x7f022c005630, mode=, block=,
>>readbytes=) at mod_logio.c:165
>> #12 0x565098d45b9a in bio_filter_in_read (bio=0x7f024800b460,
>>in=0x7f02480492a3 "", inlen=5) at ssl_engine_io.c:483
>> #13 0x7f02736b014c in BIO_read ()
>>   from /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.0
>> #14 0x7f0273a13c92 in ?? () from
>> /usr/lib/x86_64-linux-gnu/libssl.so.1.0.0
>> #15 0x7f0273a1548d in ?? () from
>> /usr/lib/x86_64-linux-gnu/libssl.so.1.0.0
>> #16 0x7f0273a12024 in ?? () from
>> /usr/lib/x86_64-linux-gnu/libssl.so.1.0.0
>> #17 0x565098d469f3 in ssl_io_input_read (inctx=0x7f022c0035b8,
>>buf=0x7f022c003600 "GET
>> /media/image/07/3c/c2/4612_13721_160192280322.jpeg HTTP/1.1\r\nHost:
>> www.onlinestore.it\r\nUser-Agent: curl/7.15+ (x64-criteo) libcurl/7.15+
>> OpenSSL zlib libidn\r\nAccept: */*\r\n\r\n", len=0x7f02627a8b38)
>>at ssl_engine_io.c:614
>> #18 0x565098d493ec in ssl_io_filter_input (f=0x7f022c005608,
>>bb=0x7f022c006dd0, mode=, block=,
>>readbytes=8000) at ssl_engine_io.c:1474
>> #19 0x565098d5cd85 in h2_filter_core_input (f=0x7f022c005980,
>>brigade=0x7f022c006dd0, mode=738225624, block=APR_NONBLOCK_READ,
>>readbytes=8000) at h2_filter.c:149
>> #20 0x565098d6809b in h2_session_read (session=0x7f022c006920,
>>block=) at h2_session.c:1440
>> #21 0x565098d6b97f in h2_session_process (session=0x7f022c006920,
>>async=3964) at h2_session.c:2074
>> #22 0x565098d5b84e in h2_conn_run (ctx=,
>> c=0x7f025c03f7d8)
>>at h2_conn.c:218
>> #23 0x565098d5e551 in h2_h2_process_conn (c=0x7f025c03f7d8) at
>> h2_h2.c:658
>> #24 0x565098d00fd0 in ap_run_process_connection (c=0x7f025c03f7d8)
>>at connection.c:42
>> #25 0x565098d9b6d0 in process_socket (my_thread_num=,
>>my_child_num=, cs=0x7f025c03f748, sock=,
>>p=, thd=) at event.c:1134
>> #26 worker_thread (thd=0xf5e, dummy=0xf7c) at event.c:2137
>> #27 0x7f02728680a4 in start_thread ()
>>   from /lib/x86_64-linux-gnu/libpthread.so.0
>> #28 0x7f027259d62d in clone () from /lib/x86_64-linux-gnu/libc.so.6
>> (gdb) (gdb) quit
>> Reading symbols from /usr/local/apache/bin/httpd...Reading symbols from
>> /usr/lib/debug//usr/local/apache2/bin/httpd...done.
>> done.
>> [Thread debugging using libthread_db enabled]
>> Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
>> Core was generated by `/usr/local/apache/bin/httpd -DFOREGROUND

Re: release v1.9.2

[Stefan Priebe - Profihost AG]

t_bucket_read (a=0x7f0234000b08,
str=0x7f0260fa57b8, len=0x7f0260fa57c0, block=)
at buckets/apr_buckets_socket.c:34
#10 0x565098cf1fb1 in ap_core_input_filter (f=0x7f02340056b0,
b=0x7f0234005630, mode=, block=APR_NONBLOCK_READ,
readbytes=5) at core_filters.c:235
#11 0x565098d3aaca in logio_in_filter (f=,
bb=0x7f0234005630, mode=, block=,
readbytes=) at mod_logio.c:165
#12 0x565098d45b9a in bio_filter_in_read (bio=0x7f022801ff50,
in=0x7f02280345d3 "(\002\177", inlen=5) at ssl_engine_io.c:483
#13 0x7f02736b014c in BIO_read ()
   from /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.0
#14 0x7f0273a13c92 in ?? () from
/usr/lib/x86_64-linux-gnu/libssl.so.1.0.0
#15 0x7f0273a1548d in ?? () from
/usr/lib/x86_64-linux-gnu/libssl.so.1.0.0
#16 0x7f0273a12024 in ?? () from
/usr/lib/x86_64-linux-gnu/libssl.so.1.0.0
#17 0x565098d469f3 in ssl_io_input_read (inctx=0x7f02340035b8,
buf=0x7f0234003600 "", len=0x7f0260fa5b38) at ssl_engine_io.c:614
#18 0x565098d493ec in ssl_io_filter_input (f=0x7f0234005608,
bb=0x7f023400cd00, mode=, block=,
readbytes=8000) at ssl_engine_io.c:1474
#19 0x565098d5cd85 in h2_filter_core_input (f=0x7f0234005980,
brigade=0x7f023400cd00, mode=872467720, block=APR_NONBLOCK_READ,
readbytes=8000) at h2_filter.c:149
#20 0x565098d6809b in h2_session_read (session=0x7f023400c850,
block=) at h2_session.c:1440
#21 0x565098d6b97f in h2_session_process (session=0x7f023400c850,
async=4029) at h2_session.c:2074
#22 0x565098d5b84e in h2_conn_run (ctx=,
c=0x7f025c03d738)
at h2_conn.c:218
#23 0x565098d5e551 in h2_h2_process_conn (c=0x7f025c03d738) at
h2_h2.c:658
#24 0x565098d00fd0 in ap_run_process_connection (c=0x7f025c03d738)
at connection.c:42
#25 0x565098d9b6d0 in process_socket (my_thread_num=,
my_child_num=, cs=0x7f025c03d6a8, sock=,
p=, thd=) at event.c:1134
#26 worker_thread (thd=0xf9c, dummy=0xfbd) at event.c:2137
#27 0x7f02728680a4 in start_thread ()
   from /lib/x86_64-linux-gnu/libpthread.so.0
#28 0x7f027259d62d in clone () from /lib/x86_64-linux-gnu/libc.so.6

Stefan

Am 27.02.2017 um 19:44 schrieb Stefan Eissing:
> Damnit. Can you try the attached patch on top of mod_http2 v1.9.2? This one 
> sets a mutex on the main connection allocator if missing and is pretty close 
> to the version we ran successfully with on your site for days.
> 
> Thanks again!
> 
> -Stefan
> 
> 
> 
> 
>> Am 27.02.2017 um 16:22 schrieb Stefan Priebe - Profihost AG 
>> <s.pri...@profihost.ag>:
>>
>> Hi Stefan,
>>
>> two new crashes here.
>>
>> Reading symbols from /usr/local/apache/bin/httpd...Reading symbols from
>> /usr/lib/debug//usr/local/apache2/bin/httpd...done.
>> done.
>> (gdb) (gdb) quit
>> Reading symbols from /usr/local/apache/bin/httpd...Reading symbols from
>> /usr/lib/debug//usr/local/apache2/bin/httpd...done.
>> done.
>> [Thread debugging using libthread_db enabled]
>> Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
>> Core was generated by `/usr/local/apache/bin/httpd -DFOREGROUND'.
>> Program terminated with signal SIGSEGV, Segmentation fault.
>> #0  allocator_free (node=0x0, allocator=0x7f458005a320)
>>at memory/unix/apr_pools.c:381
>> #0  allocator_free (node=0x0, allocator=0x7f458005a320)
>>at memory/unix/apr_pools.c:381
>> #1  apr_pool_clear (pool=0x7f458004bec8) at memory/unix/apr_pools.c:793
>> #2  0x560ce83e5718 in ap_push_pool (queue_info=0x0,
>>pool_to_recycle=0x7f458005a328) at fdqueue.c:234
>> #3  0x560ce83e0a5a in process_lingering_close (cs=0x7f458004c158,
>>pfd=0x560ce8b8dda8) at event.c:1513
>> #4  0x560ce83e4590 in listener_thread (thd=0x0, dummy=0x5497f5242585c)
>>at event.c:1837
>> #5  0x7f45967610a4 in start_thread ()
>>   from /lib/x86_64-linux-gnu/libpthread.so.0
>> #6  0x7f459649662d in clone () from /lib/x86_64-linux-gnu/libc.so.6
>> (gdb) (gdb) quit
>>
>>
>> Reading symbols from /usr/local/apache/bin/httpd...Reading symbols from
>> /usr/lib/debug//usr/local/apache2/bin/httpd...done.
>> done.
>> [Thread debugging using libthread_db enabled]
>> Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
>> Core was generated by `/usr/local/apache/bin/httpd -DFOREGROUND'.
>> Program terminated with signal SIGSEGV, Segmentation fault.
>> #0  allocator_free (node=0x0, allocator=0x7f4580062840)
>>at memory/unix/apr_pools.c:381
>> #0  allocator_free (node=0x0, allocator=0x7f4580062840)
>>at memory/unix/apr_pools.c:381
>> #1  apr_pool_clear (pool=0x7f4580069188) at memory/unix/apr_pools.c:793
>> #2  0x560ce83

Re: release v1.9.2

[Stefan Priebe - Profihost AG]

Hi Stefan,

two new crashes here.

Reading symbols from /usr/local/apache/bin/httpd...Reading symbols from
/usr/lib/debug//usr/local/apache2/bin/httpd...done.
done.
(gdb) (gdb) quit
Reading symbols from /usr/local/apache/bin/httpd...Reading symbols from
/usr/lib/debug//usr/local/apache2/bin/httpd...done.
done.
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/usr/local/apache/bin/httpd -DFOREGROUND'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  allocator_free (node=0x0, allocator=0x7f458005a320)
at memory/unix/apr_pools.c:381
#0  allocator_free (node=0x0, allocator=0x7f458005a320)
at memory/unix/apr_pools.c:381
#1  apr_pool_clear (pool=0x7f458004bec8) at memory/unix/apr_pools.c:793
#2  0x560ce83e5718 in ap_push_pool (queue_info=0x0,
pool_to_recycle=0x7f458005a328) at fdqueue.c:234
#3  0x560ce83e0a5a in process_lingering_close (cs=0x7f458004c158,
pfd=0x560ce8b8dda8) at event.c:1513
#4  0x560ce83e4590 in listener_thread (thd=0x0, dummy=0x5497f5242585c)
at event.c:1837
#5  0x7f45967610a4 in start_thread ()
   from /lib/x86_64-linux-gnu/libpthread.so.0
#6  0x7f459649662d in clone () from /lib/x86_64-linux-gnu/libc.so.6
(gdb) (gdb) quit


Reading symbols from /usr/local/apache/bin/httpd...Reading symbols from
/usr/lib/debug//usr/local/apache2/bin/httpd...done.
done.
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/usr/local/apache/bin/httpd -DFOREGROUND'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  allocator_free (node=0x0, allocator=0x7f4580062840)
at memory/unix/apr_pools.c:381
#0  allocator_free (node=0x0, allocator=0x7f4580062840)
at memory/unix/apr_pools.c:381
#1  apr_pool_clear (pool=0x7f4580069188) at memory/unix/apr_pools.c:793
#2  0x560ce83e5718 in ap_push_pool (queue_info=0x0,
pool_to_recycle=0x7f4580062848) at fdqueue.c:234
#3  0x560ce83e0a5a in process_lingering_close (cs=0x7f4580069418,
pfd=0x560ce8b8dda8) at event.c:1513
#4  0x560ce83e4590 in listener_thread (thd=0x0, dummy=0x549845279ce62)
at event.c:1837
#5  0x7f45967610a4 in start_thread ()
   from /lib/x86_64-linux-gnu/libpthread.so.0
#6  0x7f459649662d in clone () from /lib/x86_64-linux-gnu/libc.so.6
(gdb) (gdb) quit

Stefan
Am 26.02.2017 um 19:46 schrieb Stefan Priebe - Profihost AG:
> Hi Stefan,
> 
> currently everything fine. No segfaults.
> 
> Stefan
> 
> Am 25.02.2017 um 20:40 schrieb Stefan Priebe - Profihost AG:
>> Hi Stefan,
>> Am 25.02.2017 um 13:51 schrieb Stefan Eissing:
>>> Stefan,
>>>
>>> whenever you have time, please deploy 
>>> https://github.com/icing/mod_h2/releases/tag/v1.9.2
>>>
>>> I added an own allocator with mutex to the http/2 main session. That is 
>>> something of a middle-ground between placing that on the main conn (as we 
>>> had in the crash free version) and 1.9.1 behaviour. Thanks!
>>
>> done. But please keep in mind that this crash might be very rare and
>> might even have happened with v1.9.0 if we've waited more time.
>>
>> Greets,
>> Stefan
>>
>>> -Stefan
>>>
>>>> Am 24.02.2017 um 19:31 schrieb Stefan Priebe - Profihost AG 
>>>> <s.pri...@profihost.ag>:
>>>>
>>>> Hi Yann,
>>>>
>>>> here we go:
>>>>
>>>> (gdb) p *ipsub
>>>> $1 = {family = 2, sub = {1367753145, 0, 0, 0}, mask = {4294967295,
>>>> 4294967295, 4294967295, 4294967295}}
>>>>
>>>> (gdb) p *sa
>>>> $2 = {pool = 0x10b500c4ff0b0a, hostname = 0x503040203030102 >>> Cannot access memory at address 0x503040203030102>,
>>>>  servname = 0x17d01040405 >>> 0x17d01040405>, port = 770, family = 554829073,
>>>>  salen = 319177009, ipaddr_len = 570909009, addr_str_len = -2127424399,
>>>> ipaddr_ptr = 0x24f0d15215c1b142,
>>>>  next = 0x17160a0982726233, sa = {sin = {sin_family = 6424, sin_port =
>>>> 9498, sin_addr = {s_addr = 690497318},
>>>>  sin_zero = "*456789:"}, sin6 = {sin6_family = 6424, sin6_port =
>>>> 9498, sin6_flowinfo = 690497318, sin6_addr = {__in6_u = {
>>>>  __u6_addr8 = "*456789:CDEFGHIJ", __u6_addr16 = {13354, 13877,
>>>> 14391, 14905, 17475, 17989, 18503, 19017}, __u6_addr32 = {
>>>>909456426, 976828471, 1178944579, 1246316615}}},
>>>> sin6_scope_id = 1448432723}, sas = {ss_family = 6424,
>>>>  __ss_align = 4195446337656140842,
>>>>  __ss_padding =
>>>&

Re: release v1.9.2

[Stefan Priebe - Profihost AG]

Hi Stefan,

currently everything fine. No segfaults.

Stefan

Am 25.02.2017 um 20:40 schrieb Stefan Priebe - Profihost AG:
> Hi Stefan,
> Am 25.02.2017 um 13:51 schrieb Stefan Eissing:
>> Stefan,
>>
>> whenever you have time, please deploy 
>> https://github.com/icing/mod_h2/releases/tag/v1.9.2
>>
>> I added an own allocator with mutex to the http/2 main session. That is 
>> something of a middle-ground between placing that on the main conn (as we 
>> had in the crash free version) and 1.9.1 behaviour. Thanks!
> 
> done. But please keep in mind that this crash might be very rare and
> might even have happened with v1.9.0 if we've waited more time.
> 
> Greets,
> Stefan
> 
>> -Stefan
>>
>>> Am 24.02.2017 um 19:31 schrieb Stefan Priebe - Profihost AG 
>>> <s.pri...@profihost.ag>:
>>>
>>> Hi Yann,
>>>
>>> here we go:
>>>
>>> (gdb) p *ipsub
>>> $1 = {family = 2, sub = {1367753145, 0, 0, 0}, mask = {4294967295,
>>> 4294967295, 4294967295, 4294967295}}
>>>
>>> (gdb) p *sa
>>> $2 = {pool = 0x10b500c4ff0b0a, hostname = 0x503040203030102 >> Cannot access memory at address 0x503040203030102>,
>>>  servname = 0x17d01040405 >> 0x17d01040405>, port = 770, family = 554829073,
>>>  salen = 319177009, ipaddr_len = 570909009, addr_str_len = -2127424399,
>>> ipaddr_ptr = 0x24f0d15215c1b142,
>>>  next = 0x17160a0982726233, sa = {sin = {sin_family = 6424, sin_port =
>>> 9498, sin_addr = {s_addr = 690497318},
>>>  sin_zero = "*456789:"}, sin6 = {sin6_family = 6424, sin6_port =
>>> 9498, sin6_flowinfo = 690497318, sin6_addr = {__in6_u = {
>>>  __u6_addr8 = "*456789:CDEFGHIJ", __u6_addr16 = {13354, 13877,
>>> 14391, 14905, 17475, 17989, 18503, 19017}, __u6_addr32 = {
>>>909456426, 976828471, 1178944579, 1246316615}}},
>>> sin6_scope_id = 1448432723}, sas = {ss_family = 6424,
>>>  __ss_align = 4195446337656140842,
>>>  __ss_padding =
>>> "CDEFGHIJSTUVWXYZcdefghijstuvwxyz\203\204\205\206\207\210\211\212\222\223\224\225\226\227\230\231\232\242\243\244\245\246\247\250\251\252\262\263\264\265\266\267\270\271\272\302\303\304\305\306\307\310\311\312\322\323\324\325\326\327\330\331\332\341\342\343\344\345\346\347\350\351\352\361\362\363\364\365\366\367\370\371\372\377\304\000\037\001\000\003"}}}
>>>
>>> (gdb) p *(struct in6_addr *)sa
>>> $3 = {__in6_u = {__u6_addr8 =
>>> "\n\v\377\304\000\265\020\000\002\001\003\003\002\004\003\005",
>>> __u6_addr16 = {2826, 50431, 46336,
>>>  16, 258, 771, 1026, 1283}, __u6_addr32 = {3305048842, 1094912,
>>> 50528514, 84083714}}}
>>>
>>>
>>> Stefan
>>>
>>> Am 24.02.2017 um 14:18 schrieb Yann Ylavic:
>>>> Hi Stefan (Priebe),
>>>>
>>>> Is IPv6 (really) involved in your network?
>>>>
>>>> Could you please show up the gdb output of the below ?
>>>>
>>>> On Fri, Feb 24, 2017 at 2:07 PM, Yann Ylavic <ylavic@gmail.com> wrote:
>>>>>
>>>>> 1078 APR_DECLARE(int) apr_ipsubnet_test(apr_ipsubnet_t *ipsub,
>>>>> apr_sockaddr_t *sa)
>>>>> 1079 {
>>>>> 1080 #if APR_HAVE_IPV6
>>>>> 1081 /* XXX This line will segv on Win32 build with APR_HAVE_IPV6,
>>>>> 1082  * but without the IPV6 drivers installed.
>>>>> 1083  */
>>>>> 1084 if (sa->family == AF_INET) {
>>>>> 1085 if (ipsub->family == AF_INET &&
>>>>> 1086 ((sa->sa.sin.sin_addr.s_addr & ipsub->mask[0]) ==
>>>>> ipsub->sub[0])) {
>>>>> 1087 return 1;
>>>>> 1088 }
>>>>> 1089 }
>>>>> 1090 else if (IN6_IS_ADDR_V4MAPPED((struct in6_addr 
>>>>> *)sa->ipaddr_ptr)) {
>>>>> 1091 if (ipsub->family == AF_INET &&
>>>>> 1092 (((apr_uint32_t *)sa->ipaddr_ptr)[3] &
>>>>> ipsub->mask[0]) == ipsub->sub[0]) {
>>>>> 1093 return 1;
>>>>> 1094 }
>>>>> 1095 }
>>>>
>>>> (gdb) p *ipsub
>>>> (gdb) p *sa
>>>> (gdb) p *(struct in6_addr *)sa
>>>>
>>>> and possibly more to come...
>>>>
>>>>
>>>> Thanks,
>>>> Yann.
>>>>
>>
>> Stefan Eissing
>>
>> bytes GmbH
>> Hafenstrasse 16
>> 48155 Münster
>> www.greenbytes.de
>>

Re: release v1.9.0

[Stefan Priebe - Profihost AG]

Hi Stefan,
Am 25.02.2017 um 13:51 schrieb Stefan Eissing:
> Stefan,
> 
> whenever you have time, please deploy 
> https://github.com/icing/mod_h2/releases/tag/v1.9.2
> 
> I added an own allocator with mutex to the http/2 main session. That is 
> something of a middle-ground between placing that on the main conn (as we had 
> in the crash free version) and 1.9.1 behaviour. Thanks!

done. But please keep in mind that this crash might be very rare and
might even have happened with v1.9.0 if we've waited more time.

Greets,
Stefan

> -Stefan
> 
>> Am 24.02.2017 um 19:31 schrieb Stefan Priebe - Profihost AG 
>> <s.pri...@profihost.ag>:
>>
>> Hi Yann,
>>
>> here we go:
>>
>> (gdb) p *ipsub
>> $1 = {family = 2, sub = {1367753145, 0, 0, 0}, mask = {4294967295,
>> 4294967295, 4294967295, 4294967295}}
>>
>> (gdb) p *sa
>> $2 = {pool = 0x10b500c4ff0b0a, hostname = 0x503040203030102 > Cannot access memory at address 0x503040203030102>,
>>  servname = 0x17d01040405 > 0x17d01040405>, port = 770, family = 554829073,
>>  salen = 319177009, ipaddr_len = 570909009, addr_str_len = -2127424399,
>> ipaddr_ptr = 0x24f0d15215c1b142,
>>  next = 0x17160a0982726233, sa = {sin = {sin_family = 6424, sin_port =
>> 9498, sin_addr = {s_addr = 690497318},
>>  sin_zero = "*456789:"}, sin6 = {sin6_family = 6424, sin6_port =
>> 9498, sin6_flowinfo = 690497318, sin6_addr = {__in6_u = {
>>  __u6_addr8 = "*456789:CDEFGHIJ", __u6_addr16 = {13354, 13877,
>> 14391, 14905, 17475, 17989, 18503, 19017}, __u6_addr32 = {
>>909456426, 976828471, 1178944579, 1246316615}}},
>> sin6_scope_id = 1448432723}, sas = {ss_family = 6424,
>>  __ss_align = 4195446337656140842,
>>  __ss_padding =
>> "CDEFGHIJSTUVWXYZcdefghijstuvwxyz\203\204\205\206\207\210\211\212\222\223\224\225\226\227\230\231\232\242\243\244\245\246\247\250\251\252\262\263\264\265\266\267\270\271\272\302\303\304\305\306\307\310\311\312\322\323\324\325\326\327\330\331\332\341\342\343\344\345\346\347\350\351\352\361\362\363\364\365\366\367\370\371\372\377\304\000\037\001\000\003"}}}
>>
>> (gdb) p *(struct in6_addr *)sa
>> $3 = {__in6_u = {__u6_addr8 =
>> "\n\v\377\304\000\265\020\000\002\001\003\003\002\004\003\005",
>> __u6_addr16 = {2826, 50431, 46336,
>>  16, 258, 771, 1026, 1283}, __u6_addr32 = {3305048842, 1094912,
>> 50528514, 84083714}}}
>>
>>
>> Stefan
>>
>> Am 24.02.2017 um 14:18 schrieb Yann Ylavic:
>>> Hi Stefan (Priebe),
>>>
>>> Is IPv6 (really) involved in your network?
>>>
>>> Could you please show up the gdb output of the below ?
>>>
>>> On Fri, Feb 24, 2017 at 2:07 PM, Yann Ylavic <ylavic@gmail.com> wrote:
>>>>
>>>> 1078 APR_DECLARE(int) apr_ipsubnet_test(apr_ipsubnet_t *ipsub,
>>>> apr_sockaddr_t *sa)
>>>> 1079 {
>>>> 1080 #if APR_HAVE_IPV6
>>>> 1081 /* XXX This line will segv on Win32 build with APR_HAVE_IPV6,
>>>> 1082  * but without the IPV6 drivers installed.
>>>> 1083  */
>>>> 1084 if (sa->family == AF_INET) {
>>>> 1085 if (ipsub->family == AF_INET &&
>>>> 1086 ((sa->sa.sin.sin_addr.s_addr & ipsub->mask[0]) ==
>>>> ipsub->sub[0])) {
>>>> 1087 return 1;
>>>> 1088 }
>>>> 1089 }
>>>> 1090 else if (IN6_IS_ADDR_V4MAPPED((struct in6_addr *)sa->ipaddr_ptr)) 
>>>> {
>>>> 1091 if (ipsub->family == AF_INET &&
>>>> 1092 (((apr_uint32_t *)sa->ipaddr_ptr)[3] &
>>>> ipsub->mask[0]) == ipsub->sub[0]) {
>>>> 1093 return 1;
>>>> 1094 }
>>>> 1095 }
>>>
>>> (gdb) p *ipsub
>>> (gdb) p *sa
>>> (gdb) p *(struct in6_addr *)sa
>>>
>>> and possibly more to come...
>>>
>>>
>>> Thanks,
>>> Yann.
>>>
> 
> Stefan Eissing
> 
> bytes GmbH
> Hafenstrasse 16
> 48155 Münster
> www.greenbytes.de
> 

Re: release v1.9.0

[Stefan Priebe - Profihost AG]

Hi Yann,

here we go:

(gdb) p *ipsub
$1 = {family = 2, sub = {1367753145, 0, 0, 0}, mask = {4294967295,
4294967295, 4294967295, 4294967295}}

(gdb) p *sa
$2 = {pool = 0x10b500c4ff0b0a, hostname = 0x503040203030102 ,
  servname = 0x17d01040405 , port = 770, family = 554829073,
  salen = 319177009, ipaddr_len = 570909009, addr_str_len = -2127424399,
ipaddr_ptr = 0x24f0d15215c1b142,
  next = 0x17160a0982726233, sa = {sin = {sin_family = 6424, sin_port =
9498, sin_addr = {s_addr = 690497318},
  sin_zero = "*456789:"}, sin6 = {sin6_family = 6424, sin6_port =
9498, sin6_flowinfo = 690497318, sin6_addr = {__in6_u = {
  __u6_addr8 = "*456789:CDEFGHIJ", __u6_addr16 = {13354, 13877,
14391, 14905, 17475, 17989, 18503, 19017}, __u6_addr32 = {
909456426, 976828471, 1178944579, 1246316615}}},
sin6_scope_id = 1448432723}, sas = {ss_family = 6424,
  __ss_align = 4195446337656140842,
  __ss_padding =
"CDEFGHIJSTUVWXYZcdefghijstuvwxyz\203\204\205\206\207\210\211\212\222\223\224\225\226\227\230\231\232\242\243\244\245\246\247\250\251\252\262\263\264\265\266\267\270\271\272\302\303\304\305\306\307\310\311\312\322\323\324\325\326\327\330\331\332\341\342\343\344\345\346\347\350\351\352\361\362\363\364\365\366\367\370\371\372\377\304\000\037\001\000\003"}}}

(gdb) p *(struct in6_addr *)sa
$3 = {__in6_u = {__u6_addr8 =
"\n\v\377\304\000\265\020\000\002\001\003\003\002\004\003\005",
__u6_addr16 = {2826, 50431, 46336,
  16, 258, 771, 1026, 1283}, __u6_addr32 = {3305048842, 1094912,
50528514, 84083714}}}


Stefan

Am 24.02.2017 um 14:18 schrieb Yann Ylavic:
> Hi Stefan (Priebe),
> 
> Is IPv6 (really) involved in your network?
> 
> Could you please show up the gdb output of the below ?
> 
> On Fri, Feb 24, 2017 at 2:07 PM, Yann Ylavic  wrote:
>>
>> 1078 APR_DECLARE(int) apr_ipsubnet_test(apr_ipsubnet_t *ipsub,
>> apr_sockaddr_t *sa)
>> 1079 {
>> 1080 #if APR_HAVE_IPV6
>> 1081 /* XXX This line will segv on Win32 build with APR_HAVE_IPV6,
>> 1082  * but without the IPV6 drivers installed.
>> 1083  */
>> 1084 if (sa->family == AF_INET) {
>> 1085 if (ipsub->family == AF_INET &&
>> 1086 ((sa->sa.sin.sin_addr.s_addr & ipsub->mask[0]) ==
>> ipsub->sub[0])) {
>> 1087 return 1;
>> 1088 }
>> 1089 }
>> 1090 else if (IN6_IS_ADDR_V4MAPPED((struct in6_addr *)sa->ipaddr_ptr)) {
>> 1091 if (ipsub->family == AF_INET &&
>> 1092 (((apr_uint32_t *)sa->ipaddr_ptr)[3] &
>> ipsub->mask[0]) == ipsub->sub[0]) {
>> 1093 return 1;
>> 1094 }
>> 1095 }
> 
> (gdb) p *ipsub
> (gdb) p *sa
> (gdb) p *(struct in6_addr *)sa
> 
> and possibly more to come...
> 
> 
> Thanks,
> Yann.
> 

Re: release v1.9.0

[Stefan Priebe - Profihost AG]

No we don't use IPv6 at all. Do you still need those values?

Greets,
Stefan

Excuse my typo sent from my mobile phone.

> Am 24.02.2017 um 14:18 schrieb Yann Ylavic :
> 
> Hi Stefan (Priebe),
> 
> Is IPv6 (really) involved in your network?
> 
> Could you please show up the gdb output of the below ?
> 
>> On Fri, Feb 24, 2017 at 2:07 PM, Yann Ylavic  wrote:
>> 
>> 1078 APR_DECLARE(int) apr_ipsubnet_test(apr_ipsubnet_t *ipsub,
>> apr_sockaddr_t *sa)
>> 1079 {
>> 1080 #if APR_HAVE_IPV6
>> 1081 /* XXX This line will segv on Win32 build with APR_HAVE_IPV6,
>> 1082  * but without the IPV6 drivers installed.
>> 1083  */
>> 1084 if (sa->family == AF_INET) {
>> 1085 if (ipsub->family == AF_INET &&
>> 1086 ((sa->sa.sin.sin_addr.s_addr & ipsub->mask[0]) ==
>> ipsub->sub[0])) {
>> 1087 return 1;
>> 1088 }
>> 1089 }
>> 1090 else if (IN6_IS_ADDR_V4MAPPED((struct in6_addr *)sa->ipaddr_ptr)) {
>> 1091 if (ipsub->family == AF_INET &&
>> 1092 (((apr_uint32_t *)sa->ipaddr_ptr)[3] &
>> ipsub->mask[0]) == ipsub->sub[0]) {
>> 1093 return 1;
>> 1094 }
>> 1095 }
> 
> (gdb) p *ipsub
> (gdb) p *sa
> (gdb) p *(struct in6_addr *)sa
> 
> and possibly more to come...
> 
> 
> Thanks,
> Yann.

Re: release v1.9.0

[Stefan Priebe - Profihost AG]

Hi Yann,
  Hi Stefan,

new trace:
Core was generated by `/usr/local/apache/bin/httpd -DFOREGROUND'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x7f459699740c in apr_ipsubnet_test (ipsub=0x7f44c003d070,
sa=0x7f4570021950) at network_io/unix/sockaddr.c:1090
#0  0x7f459699740c in apr_ipsubnet_test (ipsub=0x7f44c003d070,
sa=0x7f4570021950) at network_io/unix/sockaddr.c:1090
#1  0x7f459573b400 in find_allowdeny (r=r@entry=0x7f44c002c250,
method=method@entry=0, a=, a=)
at mod_access_compat.c:270
#2  0x7f459573b59a in check_dir_access (r=0x7f44c002c250)
at mod_access_compat.c:324
#3  0x560ce8333fd0 in ap_run_access_checker (r=0x7f44c002c250)
at request.c:87
#4  0x560ce8336ea8 in ap_process_request_internal (r=0x7f44c002c250)
at request.c:265
#5  0x560ce8377b40 in ap_process_async_request (r=0x7f44c002c250)
at http_request.c:434
#6  0x560ce8377cf0 in ap_process_request (r=0x7f44c002c250)
at http_request.c:471
#7  0x560ce83b8aad in h2_task_process_request (c=,
task=) at h2_task.c:612
#8  h2_task_process_conn (c=0x7f44c003d070) at h2_task.c:659
#9  0x560ce8347fd0 in ap_run_process_connection (c=0x7f44c0026220)
at connection.c:42
#10 0x560ce83b9eaf in h2_task_do (task=0x7f44c002a240,
thread=0x560ce8afc900, worker_id=365015362) at h2_task.c:578
#11 0x560ce83c362f in execute (thread=0x560ce8afc900,
wctx=0x560ce8afc8c0)
at h2_worker.c:46
#12 0x7f45967610a4 in start_thread ()
   from /lib/x86_64-linux-gnu/libpthread.so.0
#13 0x7f459649662d in clone () from /lib/x86_64-linux-gnu/libc.so.6

Stefan

Am 23.02.2017 um 07:59 schrieb Stefan Priebe - Profihost AG:
> Am 22.02.2017 um 12:22 schrieb Yann Ylavic:
>> Hi Stefan,
>>
>> On Wed, Feb 22, 2017 at 11:32 AM, Stefan Priebe - Profihost AG
>> <s.pri...@profihost.ag> wrote:
>>>
>>> @Yann how should i test? Vanilla 2.4.25 + MPM V7 + mod_http2 v1.9.1?
>>
>> Yes, I think this is the right thing to do for now (no more patches than v7).
>>
>>> Or
>>> do i need V8 or something else?
>>
>> Not ready yet, I'll propose it when that's the case if you can test it then.
>> That's an mpm_event optimization (hopefully) only, v7 is good from
>> correctness POV...
> 
> OK it's running. Will report back.
> 
> Greets,
> Stefan
> 
>> Thanks for testing, still!
>>
>> Regards,
>> Yann.
>>

Re: release v1.9.0

[Stefan Priebe - Profihost AG]

Am 22.02.2017 um 12:22 schrieb Yann Ylavic:
> Hi Stefan,
> 
> On Wed, Feb 22, 2017 at 11:32 AM, Stefan Priebe - Profihost AG
> <s.pri...@profihost.ag> wrote:
>>
>> @Yann how should i test? Vanilla 2.4.25 + MPM V7 + mod_http2 v1.9.1?
> 
> Yes, I think this is the right thing to do for now (no more patches than v7).
> 
>> Or
>> do i need V8 or something else?
> 
> Not ready yet, I'll propose it when that's the case if you can test it then.
> That's an mpm_event optimization (hopefully) only, v7 is good from
> correctness POV...

OK it's running. Will report back.

Greets,
Stefan

> Thanks for testing, still!
> 
> Regards,
> Yann.
> 

Re: release v1.9.0

[Stefan Priebe - Profihost AG]

Hi Stefan,
  Hi Yann,

thanks for v1.9.1 i'm happy to test.

@Yann how should i test? Vanilla 2.4.25 + MPM V7 + mod_http2 v1.9.1? Or
do i need V8 or something else?

Greets,
Stefan

Am 22.02.2017 um 11:31 schrieb Stefan Eissing:
> v1.9.1 is out. Please test at your leisure.
> 
>> Am 21.02.2017 um 09:40 schrieb Stefan Priebe - Profihost AG 
>> <s.pri...@profihost.ag>:
>>
>> Hi Yann,
>>
>> Am 20.02.2017 um 16:38 schrieb Yann Ylavic:
>>> On Wed, Feb 15, 2017 at 8:53 PM, Stefan Priebe - Profihost AG
>>> <s.pri...@profihost.ag> wrote:
>>>>
>>>> still no segfaults.
>>>
>>> Great!
>>>
>>>>
>>>> @Yann
>>>> Are those patches (the addon on top of v7) and the one on top of mod_ssl
>>>> still correct / needed?
>>>
>>> I think so, but maybe I'm a bit lost (see below)...
>>>
>>>>
>>>> Am 15.02.2017 um 12:45 schrieb Stefan Priebe - Profihost AG:
>>>>>
>>>>> Am 15.02.2017 um 12:19 schrieb Yann Ylavic:
>>>>>>
>>>>>> Is this with or without the mpm_event's wakeup and/or allocator patches?
>>>>>
>>>>> it's with the mpm_event_listener_wakeup_bug57399_V7 +
>>>
>>> Does this includes any change besides v7 from bugzilla?
>>
>> Yes but just the ones mentioned below. I think i'll wait for v1.9.1 +
>> MPM v8 which may include your patch for mod_http2 as well? Stefan?
>>
>> Stefan
>>
>>>
>>> Also finally... I really wish we had something like v6 in mpm_event,
>>> these locks around pollset operations seem really unnecessary to me
>>> (and likely not good performance wise).
>>> I think the (very unlikely) race mentioned in
>>> https://svn.apache.org/r1779354 could be addressed in the listener
>>> itself (while processing the queues, lock held) rather than every
>>> worker.
>>>
>>> I you could try the v8 I'll try to propose soon it would be really
>>> nice of you (as usual ;)
>>>
>>>>>
>>>>> --- a/build/httpd/server/mpm/event/event.c  (revision 1776076)
>>>>> +++ b/build/httpd/server/mpm/event/event.c  (working copy)
>>>
>>> This one is needed I think, I was waiting for your feedbacks since it
>>> mainly affects http2.
>>> Everything looking good, I just committed it to trunk (r1783755), the
>>> final patch would be [1].
>>>
>>> I also committed the corresponding changes in mod_http2 (r1783756)
>>> which don't seem to be in v1.9.0, so you may need [2] and [3] too.
>>>
>>>>>
>>>>> Index: a/build/httpd/modules/ssl/ssl_engine_io.c
>>>>> ===
>>>>> --- a/build/httpd/modules/ssl/ssl_engine_io.c (revision 1781324)
>>>>> +++ b/build/httpd/modules/ssl/ssl_engine_io.c (working copy)
>>>
>>> This one is in trunk already (r1781582), but without this change:
>>>
>>>>> -if (APR_BUCKET_IS_METADATA(bucket)) {
>>>>> +if (APR_BUCKET_IS_METADATA(bucket) || !filter_ctx->pssl) {
>>>
>>> So I'd suggest to use [4] instead.
>>> No harm, though, this case cannot happen in current httpd, but as
>>> discussed in another thread we should handle it another way.
>>>
>>>
>>> To conclude, I think you should be using: httpd-2.4.25 +
>>> mod_http-v1.9.0 + PR57399-v7.patch + [1] + [2] + [3] + [4].
>>>
>>> Other than PR57399-v7, they are all in trunk now, so hopefully it will
>>> be easier to talk about them (e.g. with revision number).
>>>
>>> Regards,
>>> Yann.
>>>
>>>
>>> [1] 
>>> http://svn.apache.org/viewvc/httpd/httpd/trunk/server/mpm/event/event.c?r1=1783755=1783754=1783755=patch
>>> (from r1783755)
>>> [2] 
>>> http://svn.apache.org/viewvc/httpd/httpd/trunk/modules/http2/h2_mplx.c?rev=1783756=1783755=1783756=diff
>>> (from r1783756)
>>> [3] 
>>> http://svn.apache.org/viewvc/httpd/httpd/trunk/modules/http2/h2_conn.c?rev=1783756=1783755=1783756=diff
>>> (from r1783756)
>>> [4] 
>>> http://svn.apache.org/viewvc/httpd/httpd/trunk/modules/ssl/ssl_engine_io.c?r1=1781581=1781582=1781582=patch
>>> (from r1781582).
>>>
>>> PS: I could not find a way for viewvc URLs above to express a single
>>> diff for a whole revision change (e.g. [2] and [3] above are two files
>>> changed in the same commit...).
>>> With the svn client, it would be simply these three diffs:
>>> [svn.1] svn diff -r 1783754:1783755
>>> http://svn.apache.org/repos/asf/httpd/httpd/trunk/
>>> [svn.2-3] svn diff -r 1783755:1783756
>>> http://svn.apache.org/repos/asf/httpd/httpd/trunk/
>>> [svn.4] svn diff -r 1781581:1781582
>>> http://svn.apache.org/repos/asf/httpd/httpd/trunk/
> 
> Stefan Eissing
> 
> bytes GmbH
> Hafenstrasse 16
> 48155 Münster
> www.greenbytes.de
> 

Re: release v1.9.0

[Stefan Priebe - Profihost AG]

Hi,

Am 16.02.2017 um 11:39 schrieb Stefan Eissing:
> 
>> Am 15.02.2017 um 20:53 schrieb Stefan Priebe - Profihost AG 
>> <s.pri...@profihost.ag>:
>>
>> Hi,
>>
>> still no segfaults.
> 
> My heart sings with joy. Can you keep on sending that message every morning? 
> thanks!

I've no time tomorrow morning - so i've just checked again. Still no
segfaults.

Greets,
Stefan

>> @Yann
>> Are those patches (the addon on top of v7) and the one on top of mod_ssl
>> still correct / needed?
>>
>> Stefan
>>
>> Am 15.02.2017 um 12:45 schrieb Stefan Priebe - Profihost AG:
>>> Hi,
>>> Am 15.02.2017 um 12:19 schrieb Yann Ylavic:
>>>> Hi Stefan,
>>>>
>>>> On Wed, Feb 15, 2017 at 9:34 AM, Stefan Priebe - Profihost AG
>>>> <s.pri...@profihost.ag> wrote:
>>>>> Current status: no segfaults.
>>>>
>>>> Is this with or without the mpm_event's wakeup and/or allocator patches?
>>>
>>> it's with the mpm_event_listener_wakeup_bug57399_V7 +
>>>
>>> --- a/build/httpd/server/mpm/event/event.c  (revision 1776076)
>>> +++ b/build/httpd/server/mpm/event/event.c  (working copy)
>>> @@ -1743,6 +1743,8 @@ static void * APR_THREAD_FUNC listener_thread(apr_
>>> enable_listensocks(process_slot);
>>> }
>>> if (!listeners_disabled) {
>>> +apr_thread_mutex_t *mutex;
>>> +
>>> lr = (ap_listen_rec *) pt->baton;
>>> ap_pop_pool(, worker_queue_info);
>>>
>>> @@ -1751,19 +1753,24 @@ static void * APR_THREAD_FUNC listener_thread(apr_
>>> apr_allocator_t *allocator;
>>>
>>> apr_allocator_create();
>>> -apr_allocator_max_free_set(allocator,
>>> -   ap_max_mem_free);
>>> -apr_pool_create_ex(, pconf, NULL,
>>> allocator);
>>> -apr_allocator_owner_set(allocator, ptrans);
>>> -if (ptrans == NULL) {
>>> +apr_allocator_max_free_set(allocator,
>>> ap_max_mem_free);
>>> +rc = apr_pool_create_ex(, pconf, NULL,
>>> +allocator);
>>> +if (rc != APR_SUCCESS) {
>>> ap_log_error(APLOG_MARK, APLOG_CRIT, rc,
>>>  ap_server_conf, APLOGNO(03097)
>>>  "Failed to create transaction
>>> pool");
>>> +apr_allocator_destroy(allocator);
>>> signal_threads(ST_GRACEFUL);
>>> return NULL;
>>> }
>>> +apr_allocator_owner_set(allocator, ptrans);
>>> +apr_pool_tag(ptrans, "transaction");
>>> }
>>> -apr_pool_tag(ptrans, "transaction");
>>> +apr_thread_mutex_create(,
>>> APR_THREAD_MUTEX_DEFAULT,
>>> +ptrans);
>>> +apr_allocator_mutex_set(apr_pool_allocator_get(ptrans),
>>> +mutex);
>>>
>>> get_worker(_idle_worker, 1, _were_busy);
>>> rc = lr->accept_func(, lr, ptrans);
>>>
>>>
>>> +
>>>
>>> Index: a/build/httpd/modules/ssl/ssl_engine_io.c
>>> ===
>>> --- a/build/httpd/modules/ssl/ssl_engine_io.c (revision 1781324)
>>> +++ b/build/httpd/modules/ssl/ssl_engine_io.c (working copy)
>>> @@ -138,6 +138,7 @@ static int bio_filter_out_pass(bio_filter_out_ctx_
>>>
>>> outctx->rc = ap_pass_brigade(outctx->filter_ctx->pOutputFilter->next,
>>>  outctx->bb);
>>> +apr_brigade_cleanup(outctx->bb);
>>> /* Fail if the connection was reset: */
>>> if (outctx->rc == APR_SUCCESS && outctx->c->aborted) {
>>> outctx->rc = APR_ECONNRESET;
>>> @@ -1699,13 +1700,12 @@ static apr_status_t ssl_io_filter_output(ap_filter
>>> while (!APR

Re: release v1.9.0

[Stefan Priebe - Profihost AG]

Hi,

still no segfaults.

@Yann
Are those patches (the addon on top of v7) and the one on top of mod_ssl
still correct / needed?

Stefan

Am 15.02.2017 um 12:45 schrieb Stefan Priebe - Profihost AG:
> Hi,
> Am 15.02.2017 um 12:19 schrieb Yann Ylavic:
>> Hi Stefan,
>>
>> On Wed, Feb 15, 2017 at 9:34 AM, Stefan Priebe - Profihost AG
>> <s.pri...@profihost.ag> wrote:
>>> Current status: no segfaults.
>>
>> Is this with or without the mpm_event's wakeup and/or allocator patches?
> 
> it's with the mpm_event_listener_wakeup_bug57399_V7 +
> 
> --- a/build/httpd/server/mpm/event/event.c  (revision 1776076)
> +++ b/build/httpd/server/mpm/event/event.c  (working copy)
> @@ -1743,6 +1743,8 @@ static void * APR_THREAD_FUNC listener_thread(apr_
>  enable_listensocks(process_slot);
>  }
>  if (!listeners_disabled) {
> +apr_thread_mutex_t *mutex;
> +
>  lr = (ap_listen_rec *) pt->baton;
>  ap_pop_pool(, worker_queue_info);
> 
> @@ -1751,19 +1753,24 @@ static void * APR_THREAD_FUNC listener_thread(apr_
>  apr_allocator_t *allocator;
> 
>  apr_allocator_create();
> -apr_allocator_max_free_set(allocator,
> -   ap_max_mem_free);
> -apr_pool_create_ex(, pconf, NULL,
> allocator);
> -apr_allocator_owner_set(allocator, ptrans);
> -if (ptrans == NULL) {
> +apr_allocator_max_free_set(allocator,
> ap_max_mem_free);
> +rc = apr_pool_create_ex(, pconf, NULL,
> +allocator);
> +if (rc != APR_SUCCESS) {
>  ap_log_error(APLOG_MARK, APLOG_CRIT, rc,
>   ap_server_conf, APLOGNO(03097)
>   "Failed to create transaction
> pool");
> +apr_allocator_destroy(allocator);
>  signal_threads(ST_GRACEFUL);
>  return NULL;
>  }
> +apr_allocator_owner_set(allocator, ptrans);
> +apr_pool_tag(ptrans, "transaction");
>  }
> -apr_pool_tag(ptrans, "transaction");
> +apr_thread_mutex_create(,
> APR_THREAD_MUTEX_DEFAULT,
> +ptrans);
> +apr_allocator_mutex_set(apr_pool_allocator_get(ptrans),
> +mutex);
> 
>  get_worker(_idle_worker, 1, _were_busy);
>  rc = lr->accept_func(, lr, ptrans);
> 
> 
> +
> 
> Index: a/build/httpd/modules/ssl/ssl_engine_io.c
> ===
> --- a/build/httpd/modules/ssl/ssl_engine_io.c (revision 1781324)
> +++ b/build/httpd/modules/ssl/ssl_engine_io.c (working copy)
> @@ -138,6 +138,7 @@ static int bio_filter_out_pass(bio_filter_out_ctx_
> 
>  outctx->rc = ap_pass_brigade(outctx->filter_ctx->pOutputFilter->next,
>   outctx->bb);
> +apr_brigade_cleanup(outctx->bb);
>  /* Fail if the connection was reset: */
>  if (outctx->rc == APR_SUCCESS && outctx->c->aborted) {
>  outctx->rc = APR_ECONNRESET;
> @@ -1699,13 +1700,12 @@ static apr_status_t ssl_io_filter_output(ap_filter
>  while (!APR_BRIGADE_EMPTY(bb) && status == APR_SUCCESS) {
>  apr_bucket *bucket = APR_BRIGADE_FIRST(bb);
> 
> -if (APR_BUCKET_IS_METADATA(bucket)) {
> +if (APR_BUCKET_IS_METADATA(bucket) || !filter_ctx->pssl) {
>  /* Pass through metadata buckets untouched.  EOC is
>   * special; terminate the SSL layer first. */
>  if (AP_BUCKET_IS_EOC(bucket)) {
>  ssl_filter_io_shutdown(filter_ctx, f->c, 0);
>  }
> -AP_DEBUG_ASSERT(APR_BRIGADE_EMPTY(outctx->bb));
> 
>  /* Metadata buckets are passed one per brigade; it might
>   * be more efficient (but also more complex) to use
> @@ -1712,11 +1712,10 @@ static apr_status_t ssl_io_filter_output(ap_filter
>   * outctx->bb as a true buffer and interleave these with
>   * data buckets. */
>  APR_BUCKET_REMOVE(bucket);
> -APR_

Re: release v1.9.0

[Stefan Priebe - Profihost AG]

Hi,
Am 15.02.2017 um 12:19 schrieb Yann Ylavic:
> Hi Stefan,
> 
> On Wed, Feb 15, 2017 at 9:34 AM, Stefan Priebe - Profihost AG
> <s.pri...@profihost.ag> wrote:
>> Current status: no segfaults.
> 
> Is this with or without the mpm_event's wakeup and/or allocator patches?

it's with the mpm_event_listener_wakeup_bug57399_V7 +

--- a/build/httpd/server/mpm/event/event.c  (revision 1776076)
+++ b/build/httpd/server/mpm/event/event.c  (working copy)
@@ -1743,6 +1743,8 @@ static void * APR_THREAD_FUNC listener_thread(apr_
 enable_listensocks(process_slot);
 }
 if (!listeners_disabled) {
+apr_thread_mutex_t *mutex;
+
 lr = (ap_listen_rec *) pt->baton;
 ap_pop_pool(, worker_queue_info);

@@ -1751,19 +1753,24 @@ static void * APR_THREAD_FUNC listener_thread(apr_
 apr_allocator_t *allocator;

 apr_allocator_create();
-apr_allocator_max_free_set(allocator,
-   ap_max_mem_free);
-apr_pool_create_ex(, pconf, NULL,
allocator);
-apr_allocator_owner_set(allocator, ptrans);
-if (ptrans == NULL) {
+apr_allocator_max_free_set(allocator,
ap_max_mem_free);
+rc = apr_pool_create_ex(, pconf, NULL,
+allocator);
+if (rc != APR_SUCCESS) {
 ap_log_error(APLOG_MARK, APLOG_CRIT, rc,
  ap_server_conf, APLOGNO(03097)
  "Failed to create transaction
pool");
+apr_allocator_destroy(allocator);
 signal_threads(ST_GRACEFUL);
 return NULL;
 }
+apr_allocator_owner_set(allocator, ptrans);
+apr_pool_tag(ptrans, "transaction");
 }
-apr_pool_tag(ptrans, "transaction");
+apr_thread_mutex_create(,
APR_THREAD_MUTEX_DEFAULT,
+ptrans);
+apr_allocator_mutex_set(apr_pool_allocator_get(ptrans),
+mutex);

 get_worker(_idle_worker, 1, _were_busy);
 rc = lr->accept_func(, lr, ptrans);


+

Index: a/build/httpd/modules/ssl/ssl_engine_io.c
===
--- a/build/httpd/modules/ssl/ssl_engine_io.c (revision 1781324)
+++ b/build/httpd/modules/ssl/ssl_engine_io.c (working copy)
@@ -138,6 +138,7 @@ static int bio_filter_out_pass(bio_filter_out_ctx_

 outctx->rc = ap_pass_brigade(outctx->filter_ctx->pOutputFilter->next,
  outctx->bb);
+apr_brigade_cleanup(outctx->bb);
 /* Fail if the connection was reset: */
 if (outctx->rc == APR_SUCCESS && outctx->c->aborted) {
 outctx->rc = APR_ECONNRESET;
@@ -1699,13 +1700,12 @@ static apr_status_t ssl_io_filter_output(ap_filter
 while (!APR_BRIGADE_EMPTY(bb) && status == APR_SUCCESS) {
 apr_bucket *bucket = APR_BRIGADE_FIRST(bb);

-if (APR_BUCKET_IS_METADATA(bucket)) {
+if (APR_BUCKET_IS_METADATA(bucket) || !filter_ctx->pssl) {
 /* Pass through metadata buckets untouched.  EOC is
  * special; terminate the SSL layer first. */
 if (AP_BUCKET_IS_EOC(bucket)) {
 ssl_filter_io_shutdown(filter_ctx, f->c, 0);
 }
-AP_DEBUG_ASSERT(APR_BRIGADE_EMPTY(outctx->bb));

 /* Metadata buckets are passed one per brigade; it might
  * be more efficient (but also more complex) to use
@@ -1712,11 +1712,10 @@ static apr_status_t ssl_io_filter_output(ap_filter
  * outctx->bb as a true buffer and interleave these with
  * data buckets. */
 APR_BUCKET_REMOVE(bucket);
-APR_BRIGADE_INSERT_HEAD(outctx->bb, bucket);
-status = ap_pass_brigade(f->next, outctx->bb);
-if (status == APR_SUCCESS && f->c->aborted)
-status = APR_ECONNRESET;
-apr_brigade_cleanup(outctx->bb);
+APR_BRIGADE_INSERT_TAIL(outctx->bb, bucket);
+if (bio_filter_out_pass(outctx) < 0) {
+status = outctx->rc;
+}
 }
 else {
 /* Filter a data bucket. */


Greets,
Stefan

Re: release v1.9.0

[Stefan Priebe - Profihost AG]

Current status: no segfaults.

Am 14.02.2017 um 19:59 schrieb Stefan Priebe - Profihost AG:
> Hi,
> 
> up and running - Thanks!
> 
> Greets,
> Stefan
> 
> Am 14.02.2017 um 17:05 schrieb Stefan Eissing:
>> Stefan,
>>
>> if you'd like, please throw
>> https://github.com/icing/mod_h2/releases/tag/v1.9.0
>> into your pit of segfaults and let's see if we find a new one!
>>
>> Many thanks for testing.
>>
>> Stefan Eissing
>>
>> bytes GmbH
>> Hafenstrasse 16
>> 48155 Münster
>> www.greenbytes.de
>>

Re: release v1.9.0

[Stefan Priebe - Profihost AG]

Hi,

up and running - Thanks!

Greets,
Stefan

Am 14.02.2017 um 17:05 schrieb Stefan Eissing:
> Stefan,
> 
> if you'd like, please throw
> https://github.com/icing/mod_h2/releases/tag/v1.9.0
> into your pit of segfaults and let's see if we find a new one!
> 
> Many thanks for testing.
> 
> Stefan Eissing
> 
> bytes GmbH
> Hafenstrasse 16
> 48155 Münster
> www.greenbytes.de
> 

Re: mod_http2 and Frequent wake-ups for mpm_event

[Stefan Priebe - Profihost AG]

Hi Stefan,

Am 09.02.2017 um 11:47 schrieb Stefan Eissing:
> Stefan,
> 
> at this point and after several efforts to write the right patch, I reached 
> the conclusion that I need to rethink the pool hierarchy and connection 
> shutdown strategy in mod_http2. The current, organically grown implementation 
> needs a refactor with the knowledge we have made so far.

OK - thanks for your help and hard work.

> So, a fix will not come today or tomorrow. But not too far away either. From 
> your comments I assume that these crashed happen not too frequently. Hope 
> this allows you to live with the current state for a while.

Sure and yes it does not happen very often.

> Btw. have the status 500 lines disappeared with the latest release? That was 
> one point still open on my list...

Yes sorry i missed to report back. It's working fine now.

> 
> Hope to give you something better to verify in your environment soon.

Just tell me i'm ready to test.

Greets,
Stefan

> Cheers,
> 
> Stefan
> 
>> Am 07.02.2017 um 20:56 schrieb Stefan Priebe - Profihost AG 
>> <s.pri...@profihost.ag>:
>>
>> Hi,
>>
>> got this one today with both patches applied:
>>
>> Core was generated by `/usr/local/apache/bin/httpd -DFOREGROUND'.
>> Program terminated with signal SIGSEGV, Segmentation fault.
>> #0  allocator_free (node=0x0, allocator=0x7f350405e030)
>>at memory/unix/apr_pools.c:381
>> #0  allocator_free (node=0x0, allocator=0x7f350405e030)
>>at memory/unix/apr_pools.c:381
>> #1  apr_pool_clear (pool=0x7f3504060468) at memory/unix/apr_pools.c:793
>> #2  0x55d11256d688 in ap_push_pool (queue_info=0x7f35040604f8,
>>pool_to_recycle=0x) at fdqueue.c:234
>> #3  0x55d11256899a in process_lingering_close (cs=0x7f3504060748,
>>pfd=0x55d1143fd7f8) at event.c:1513
>> #4  0x55d11256c4d0 in listener_thread (thd=0x7f35040604f8,
>>dummy=0x547f569acd4a6) at event.c:1837
>> #5  0x7f351757c0a4 in start_thread ()
>>   from /lib/x86_64-linux-gnu/libpthread.so.0
>> #6  0x00007f35172b162d in clone () from /lib/x86_64-linux-gnu/libc.so.6
>>
>> Am 06.02.2017 um 22:24 schrieb Stefan Priebe - Profihost AG:
>>> Am 06.02.2017 um 22:22 schrieb Stefan Priebe - Profihost AG:
>>>> Hi Stefan,
>>>>
>>>> this one does not apply on top of yann's
>>>> mpm_event_listener_wakeup_bug57399_V7.patch patch.
>>>
>>> i've now used his patch to mpm and yours for mod_http2.
>>>
>>> Stefan
>>>
>>>>
>>>> Stefan
>>>> Am 06.02.2017 um 15:34 schrieb Stefan Eissing:
>>>>> Yes, that was mixing the order. The attached v4 compiles and runs the h2 
>>>>> tests for me without errors.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>> Am 06.02.2017 um 14:43 schrieb Yann Ylavic <ylavic@gmail.com>:
>>>>>>
>>>>>> On Mon, Feb 6, 2017 at 2:31 PM, Stefan Eissing
>>>>>> <stefan.eiss...@greenbytes.de> wrote:
>>>>>>> Currently running some tests. Have crashes on the original patch in my 
>>>>>>> test suite. Fixed one, hunting for the next...
>>>>>>
>>>>>> I think it comes from my change that creates slave connections from
>>>>>> master->pool (instead of mplx's), because now slave's pool is already
>>>>>> destroyed when 
>>>>>> h2_mplx_release_and_join()->task_destroy()->h2_slave_destroy()
>>>>>> is called (hence the crash).
>>>>>>
>>>>>> I restored your original code in this new (attached) patch.
>>>>>>
>>>>>> @s.priebe, would you test this one please?
>>>>>> 
>>>>>
>>>>> Stefan Eissing
>>>>>
>>>>> bytes GmbH
>>>>> Hafenstrasse 16
>>>>> 48155 Münster
>>>>> www.greenbytes.de
>>>>>
> 
> Stefan Eissing
> 
> bytes GmbH
> Hafenstrasse 16
> 48155 Münster
> www.greenbytes.de
> 

Re: mod_http2 and Frequent wake-ups for mpm_event

[Stefan Priebe - Profihost AG]

Hi,

got this one today with both patches applied:

Core was generated by `/usr/local/apache/bin/httpd -DFOREGROUND'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  allocator_free (node=0x0, allocator=0x7f350405e030)
at memory/unix/apr_pools.c:381
#0  allocator_free (node=0x0, allocator=0x7f350405e030)
at memory/unix/apr_pools.c:381
#1  apr_pool_clear (pool=0x7f3504060468) at memory/unix/apr_pools.c:793
#2  0x55d11256d688 in ap_push_pool (queue_info=0x7f35040604f8,
pool_to_recycle=0x) at fdqueue.c:234
#3  0x55d11256899a in process_lingering_close (cs=0x7f3504060748,
pfd=0x55d1143fd7f8) at event.c:1513
#4  0x55d11256c4d0 in listener_thread (thd=0x7f35040604f8,
dummy=0x547f569acd4a6) at event.c:1837
#5  0x7f351757c0a4 in start_thread ()
   from /lib/x86_64-linux-gnu/libpthread.so.0
#6  0x7f35172b162d in clone () from /lib/x86_64-linux-gnu/libc.so.6

Am 06.02.2017 um 22:24 schrieb Stefan Priebe - Profihost AG:
> Am 06.02.2017 um 22:22 schrieb Stefan Priebe - Profihost AG:
>> Hi Stefan,
>>
>> this one does not apply on top of yann's
>> mpm_event_listener_wakeup_bug57399_V7.patch patch.
> 
> i've now used his patch to mpm and yours for mod_http2.
> 
> Stefan
> 
>>
>> Stefan
>> Am 06.02.2017 um 15:34 schrieb Stefan Eissing:
>>> Yes, that was mixing the order. The attached v4 compiles and runs the h2 
>>> tests for me without errors.
>>>
>>>
>>>
>>>
>>>
>>>> Am 06.02.2017 um 14:43 schrieb Yann Ylavic <ylavic@gmail.com>:
>>>>
>>>> On Mon, Feb 6, 2017 at 2:31 PM, Stefan Eissing
>>>> <stefan.eiss...@greenbytes.de> wrote:
>>>>> Currently running some tests. Have crashes on the original patch in my 
>>>>> test suite. Fixed one, hunting for the next...
>>>>
>>>> I think it comes from my change that creates slave connections from
>>>> master->pool (instead of mplx's), because now slave's pool is already
>>>> destroyed when 
>>>> h2_mplx_release_and_join()->task_destroy()->h2_slave_destroy()
>>>> is called (hence the crash).
>>>>
>>>> I restored your original code in this new (attached) patch.
>>>>
>>>> @s.priebe, would you test this one please?
>>>> 
>>>
>>> Stefan Eissing
>>>
>>> bytes GmbH
>>> Hafenstrasse 16
>>> 48155 Münster
>>> www.greenbytes.de
>>>

Re: mod_http2 and Frequent wake-ups for mpm_event

[Stefan Priebe - Profihost AG]

Am 06.02.2017 um 22:22 schrieb Stefan Priebe - Profihost AG:
> Hi Stefan,
> 
> this one does not apply on top of yann's
> mpm_event_listener_wakeup_bug57399_V7.patch patch.

i've now used his patch to mpm and yours for mod_http2.

Stefan

> 
> Stefan
> Am 06.02.2017 um 15:34 schrieb Stefan Eissing:
>> Yes, that was mixing the order. The attached v4 compiles and runs the h2 
>> tests for me without errors.
>>
>>
>>
>>
>>
>>> Am 06.02.2017 um 14:43 schrieb Yann Ylavic <ylavic@gmail.com>:
>>>
>>> On Mon, Feb 6, 2017 at 2:31 PM, Stefan Eissing
>>> <stefan.eiss...@greenbytes.de> wrote:
>>>> Currently running some tests. Have crashes on the original patch in my 
>>>> test suite. Fixed one, hunting for the next...
>>>
>>> I think it comes from my change that creates slave connections from
>>> master->pool (instead of mplx's), because now slave's pool is already
>>> destroyed when 
>>> h2_mplx_release_and_join()->task_destroy()->h2_slave_destroy()
>>> is called (hence the crash).
>>>
>>> I restored your original code in this new (attached) patch.
>>>
>>> @s.priebe, would you test this one please?
>>> 
>>
>> Stefan Eissing
>>
>> bytes GmbH
>> Hafenstrasse 16
>> 48155 Münster
>> www.greenbytes.de
>>

Re: mod_http2 and Frequent wake-ups for mpm_event

[Stefan Priebe - Profihost AG]

Hi Stefan,

this one does not apply on top of yann's
mpm_event_listener_wakeup_bug57399_V7.patch patch.

Stefan
Am 06.02.2017 um 15:34 schrieb Stefan Eissing:
> Yes, that was mixing the order. The attached v4 compiles and runs the h2 
> tests for me without errors.
> 
> 
> 
> 
> 
>> Am 06.02.2017 um 14:43 schrieb Yann Ylavic :
>>
>> On Mon, Feb 6, 2017 at 2:31 PM, Stefan Eissing
>>  wrote:
>>> Currently running some tests. Have crashes on the original patch in my test 
>>> suite. Fixed one, hunting for the next...
>>
>> I think it comes from my change that creates slave connections from
>> master->pool (instead of mplx's), because now slave's pool is already
>> destroyed when h2_mplx_release_and_join()->task_destroy()->h2_slave_destroy()
>> is called (hence the crash).
>>
>> I restored your original code in this new (attached) patch.
>>
>> @s.priebe, would you test this one please?
>> 
> 
> Stefan Eissing
> 
> bytes GmbH
> Hafenstrasse 16
> 48155 Münster
> www.greenbytes.de
> 

Re: mod_http2 and Frequent wake-ups for mpm_event

[Stefan Priebe - Profihost AG]

Am 06.02.2017 um 16:06 schrieb Stefan Eissing:
> It does, at the end. Traversed the directories in different order it seems.
*arg* sorry

> 
>> Am 06.02.2017 um 16:05 schrieb Stefan Priebe - Profihost AG 
>> <s.pri...@profihost.ag>:
>>
>> Am 06.02.2017 um 16:03 schrieb Stefan Eissing:
>>> I took Yann's v3 patch and changed a tiny bit in the h2. I just added a var 
>>> declaration in event.c without which it did not compile for me.
>>
>> But your patch does not contain the changes to the event mpm. That's why
>> i ask.
>>
>> Stefan
>>
>>> -Stefan
>>>
>>>> Am 06.02.2017 um 15:41 schrieb Stefan Priebe - Profihost AG 
>>>> <s.pri...@profihost.ag>:
>>>>
>>>> Hi,
>>>>
>>>> so i should test the mpm event part from Yann + your http2 part?
>>>>
>>>> Stefan
>>>>
>>>> Am 06.02.2017 um 15:34 schrieb Stefan Eissing:
>>>>> Yes, that was mixing the order. The attached v4 compiles and runs the h2 
>>>>> tests for me without errors.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>> Am 06.02.2017 um 14:43 schrieb Yann Ylavic <ylavic@gmail.com>:
>>>>>>
>>>>>> On Mon, Feb 6, 2017 at 2:31 PM, Stefan Eissing
>>>>>> <stefan.eiss...@greenbytes.de> wrote:
>>>>>>> Currently running some tests. Have crashes on the original patch in my 
>>>>>>> test suite. Fixed one, hunting for the next...
>>>>>>
>>>>>> I think it comes from my change that creates slave connections from
>>>>>> master->pool (instead of mplx's), because now slave's pool is already
>>>>>> destroyed when 
>>>>>> h2_mplx_release_and_join()->task_destroy()->h2_slave_destroy()
>>>>>> is called (hence the crash).
>>>>>>
>>>>>> I restored your original code in this new (attached) patch.
>>>>>>
>>>>>> @s.priebe, would you test this one please?
>>>>>> 
>>>>>
>>>>> Stefan Eissing
>>>>>
>>>>> bytes GmbH
>>>>> Hafenstrasse 16
>>>>> 48155 Münster
>>>>> www.greenbytes.de
>>>>>
>>>
>>> Stefan Eissing
>>>
>>> bytes GmbH
>>> Hafenstrasse 16
>>> 48155 Münster
>>> www.greenbytes.de
>>>
> 
> Stefan Eissing
> 
> bytes GmbH
> Hafenstrasse 16
> 48155 Münster
> www.greenbytes.de
> 

Re: mod_http2 and Frequent wake-ups for mpm_event

[Stefan Priebe - Profihost AG]

Am 06.02.2017 um 16:03 schrieb Stefan Eissing:
> I took Yann's v3 patch and changed a tiny bit in the h2. I just added a var 
> declaration in event.c without which it did not compile for me.

But your patch does not contain the changes to the event mpm. That's why
i ask.

Stefan

> -Stefan
> 
>> Am 06.02.2017 um 15:41 schrieb Stefan Priebe - Profihost AG 
>> <s.pri...@profihost.ag>:
>>
>> Hi,
>>
>> so i should test the mpm event part from Yann + your http2 part?
>>
>> Stefan
>>
>> Am 06.02.2017 um 15:34 schrieb Stefan Eissing:
>>> Yes, that was mixing the order. The attached v4 compiles and runs the h2 
>>> tests for me without errors.
>>>
>>>
>>>
>>>
>>>
>>>> Am 06.02.2017 um 14:43 schrieb Yann Ylavic <ylavic@gmail.com>:
>>>>
>>>> On Mon, Feb 6, 2017 at 2:31 PM, Stefan Eissing
>>>> <stefan.eiss...@greenbytes.de> wrote:
>>>>> Currently running some tests. Have crashes on the original patch in my 
>>>>> test suite. Fixed one, hunting for the next...
>>>>
>>>> I think it comes from my change that creates slave connections from
>>>> master->pool (instead of mplx's), because now slave's pool is already
>>>> destroyed when 
>>>> h2_mplx_release_and_join()->task_destroy()->h2_slave_destroy()
>>>> is called (hence the crash).
>>>>
>>>> I restored your original code in this new (attached) patch.
>>>>
>>>> @s.priebe, would you test this one please?
>>>> 
>>>
>>> Stefan Eissing
>>>
>>> bytes GmbH
>>> Hafenstrasse 16
>>> 48155 Münster
>>> www.greenbytes.de
>>>
> 
> Stefan Eissing
> 
> bytes GmbH
> Hafenstrasse 16
> 48155 Münster
> www.greenbytes.de
> 

Re: mod_http2 and Frequent wake-ups for mpm_event

[Stefan Priebe - Profihost AG]

Hi,

so i should test the mpm event part from Yann + your http2 part?

Stefan

Am 06.02.2017 um 15:34 schrieb Stefan Eissing:
> Yes, that was mixing the order. The attached v4 compiles and runs the h2 
> tests for me without errors.
> 
> 
> 
> 
> 
>> Am 06.02.2017 um 14:43 schrieb Yann Ylavic :
>>
>> On Mon, Feb 6, 2017 at 2:31 PM, Stefan Eissing
>>  wrote:
>>> Currently running some tests. Have crashes on the original patch in my test 
>>> suite. Fixed one, hunting for the next...
>>
>> I think it comes from my change that creates slave connections from
>> master->pool (instead of mplx's), because now slave's pool is already
>> destroyed when h2_mplx_release_and_join()->task_destroy()->h2_slave_destroy()
>> is called (hence the crash).
>>
>> I restored your original code in this new (attached) patch.
>>
>> @s.priebe, would you test this one please?
>> 
> 
> Stefan Eissing
> 
> bytes GmbH
> Hafenstrasse 16
> 48155 Münster
> www.greenbytes.de
> 

Re: mod_http2 and Frequent wake-ups for mpm_event

[Stefan Priebe - Profihost AG]

Am 06.02.2017 um 11:56 schrieb Yann Ylavic:
> Hi Stefan,
> 
> On Mon, Feb 6, 2017 at 9:57 AM, Stefan Priebe - Profihost AG
> <s.pri...@profihost.ag> wrote:
>>
>> your last patch results in multiple crashes every second:
> 
> Sorry about that, the changes in mpm_event were incorrect (the mutex
> was cleared with the pool when recycled, hence its pointer was
> dangling).
> 
> New patch attached, this time tested with the httpd framework (where
> the previous patch segfaulted too).

Thanks but that one had crashed already once again.

This time:

error.log:
*** Error in `/usr/local/apache/bin/httpd': free(): invalid pointer:
0x7f4bdc023dc0 ***

Core was generated by `/usr/local/apache/bin/httpd -DFOREGROUND'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  allocator_free (node=0x4480447f447e4466, allocator=0x7f4bc40008c0)
at memory/unix/apr_pools.c:381
#0  allocator_free (node=0x4480447f447e4466, allocator=0x7f4bc40008c0)
at memory/unix/apr_pools.c:381
#1  apr_pool_destroy (pool=0x7f4bc404a178) at memory/unix/apr_pools.c:856
#2  0x55dab5531bff in task_destroy (m=0x7f4c4c00e8f8,
task=0x7f4bc404e210,
called_from_master=0) at h2_mplx.c:396
#3  0x55dab5532e6b in task_done_iter (ctx=,
val=) at h2_mplx.c:1060
#4  0x7f4c8b6965e6 in apr_hash_do (
comp=comp@entry=0x55dab5545140 ,
rec=rec@entry=0x7f4c6bfe6480,
ht=) at tables/apr_hash.c:542
#5  0x55dab5545b1f in h2_ihash_iter (ih=,
fn=fn@entry=0x55dab5532e60 ,
ctx=ctx@entry=0x7f4c4c00e8f8)
at h2_util.c:315
#6  0x55dab5533433 in h2_mplx_release_and_join (m=0x7f4c4c00e8f8,
wait=0x7f4c4c00e8a0) at h2_mplx.c:615
#7  0x55dab5538ab4 in session_pool_cleanup (data=0x7f4c04005c78)
at h2_session.c:827
#8  0x7f4c8b69f48e in run_cleanups (cref=0x7f4c4c00e878)
at memory/unix/apr_pools.c:2352
#9  apr_pool_destroy (pool=0x7f4c4c00e808) at memory/unix/apr_pools.c:804
#10 0x7f4c8b69f745 in apr_pool_clear (pool=0x7f4c78058198)
at memory/unix/apr_pools.c:769
#11 0x55dab5570668 in ap_push_pool (queue_info=0x7f4bc40008c0,
pool_to_recycle=0xc4041f91) at fdqueue.c:234
#12 0x55dab556b99a in process_lingering_close (cs=0x7f4c78058478,
pfd=0x55dab6cf3fa8) at event.c:1513
#13 0x55dab556f4d0 in listener_thread (thd=0x7f4bc40008c0,
dummy=0x547dbb6874f83) at event.c:1837
#14 0x7f4c8b46e0a4 in start_thread ()
   from /lib/x86_64-linux-gnu/libpthread.so.0
#15 0x7f4c8b1a362d in clone () from /lib/x86_64-linux-gnu/libc.so.6

Greets,
Stefan

Re: mod_http2 and Frequent wake-ups for mpm_event

[Stefan Priebe - Profihost AG]

Hi Yann,

your last patch results in multiple crashes every second:
Program terminated with signal SIGSEGV, Segmentation fault.
#0  apr_pool_cleanup_kill (p=0x7f2b,
data=data@entry=0x7f2bb40478c0,
cleanup_fn=cleanup_fn@entry=0x7f2bc92285b0 ) at
memory/unix/apr_pools.c:2264
2264memory/unix/apr_pools.c: No such file or directory.
#0  apr_pool_cleanup_kill (p=0x7f2b,
data=data@entry=0x7f2bb40478c0,
cleanup_fn=cleanup_fn@entry=0x7f2bc92285b0 ) at
memory/unix/apr_pools.c:2264
#1  0x7f2bc9224871 in apr_pool_cleanup_run (p=,
data=0x7f2bb40478c0, cleanup_fn=0x7f2bc92285b0 )
at memory/unix/apr_pools.c:2342
#2  0x7f2bc9228892 in apr_socket_close (thesocket=)
at network_io/unix/sockets.c:183
#3  0x555d4f07b99a in process_lingering_close (cs=0x7f2bb4047ac8,
pfd=0x555d4f75dfa8) at event.c:1510
#4  0x555d4f07f4e0 in listener_thread (thd=0x7f2b,
dummy=0x547d8caf3534b) at event.c:1837
#5  0x7f2bc8ff20a4 in start_thread () from
/lib/x86_64-linux-gnu/libpthread.so.0
#6  0x7f2bc8d2762d in clone () from /lib/x86_64-linux-gnu/libc.so.6

Stefan

Am 06.02.2017 um 09:33 schrieb Stefan Priebe - Profihost AG:
> Hallo,
> 
> up and running (apache 2.4.25 + mpm_event V7 + mpd_http2 v1.8.11 +
> mod_ssl patch + your new allocator patch). Will report back.
> 
> Greets,
> Stefan
> 
> Am 06.02.2017 um 01:19 schrieb Yann Ylavic:
>> Hi Stefan,
>>
>> On Sun, Feb 5, 2017 at 7:51 PM, Stefan Priebe - Profihost AG
>> <s.pri...@profihost.ag> wrote:
>>>
>>> tested your patch against mod_ssl. I haven't seen any pool crashes again
>>> so it seems to fix this issue.
>>>
>>> But two new ones:
>>
>> Possibly a promising fix suggested (in another thread) by yet another
>> talented Stefan :)
>> He has spotted a possible issue in mpm_event which could affect mainly
>> mod_http2.
>> I applied his suggestion in the attached patch, and did some
>> extrapolation for similar code in mod_h2 (so all possible errors are
>> mine...).
>> Would you test this one please?
>>
>> @icing: I changed the parent pool of the slave connection from the
>> mplx pool to the master connection's (ptrans), just to simplify the
>> allocators in place for now.
>> I don't see it was needed from a concurrency POV, but if you wanted to
>> avoid locking when creating slaves' pool we can probably keep the
>> dedicated allocator finally (to reduce possible contention on
>> ptrans->mutex? No mutex needed I think since mplx doesn't seem to have
>> other subpools. Trade off with memory consumption, though).
>>
>>
>> Regards,
>> Yann.
>>

Re: mod_http2 and Frequent wake-ups for mpm_event

[Stefan Priebe - Profihost AG]

Hallo,

up and running (apache 2.4.25 + mpm_event V7 + mpd_http2 v1.8.11 +
mod_ssl patch + your new allocator patch). Will report back.

Greets,
Stefan

Am 06.02.2017 um 01:19 schrieb Yann Ylavic:
> Hi Stefan,
> 
> On Sun, Feb 5, 2017 at 7:51 PM, Stefan Priebe - Profihost AG
> <s.pri...@profihost.ag> wrote:
>>
>> tested your patch against mod_ssl. I haven't seen any pool crashes again
>> so it seems to fix this issue.
>>
>> But two new ones:
> 
> Possibly a promising fix suggested (in another thread) by yet another
> talented Stefan :)
> He has spotted a possible issue in mpm_event which could affect mainly
> mod_http2.
> I applied his suggestion in the attached patch, and did some
> extrapolation for similar code in mod_h2 (so all possible errors are
> mine...).
> Would you test this one please?
> 
> @icing: I changed the parent pool of the slave connection from the
> mplx pool to the master connection's (ptrans), just to simplify the
> allocators in place for now.
> I don't see it was needed from a concurrency POV, but if you wanted to
> avoid locking when creating slaves' pool we can probably keep the
> dedicated allocator finally (to reduce possible contention on
> ptrans->mutex? No mutex needed I think since mplx doesn't seem to have
> other subpools. Trade off with memory consumption, though).
> 
> 
> Regards,
> Yann.
> 

Re: mod_http2 and Frequent wake-ups for mpm_event

[Stefan Priebe - Profihost AG]

Hi Yann,

tested your patch against mod_ssl. I haven't seen any pool crashes again
so it seems to fix this issue.

But two new ones:

Program terminated with signal SIGABRT, Aborted.
#0  0x7ff523558067 in raise () from /lib/x86_64-linux-gnu/libc.so.6
#0  0x7ff523558067 in raise () from /lib/x86_64-linux-gnu/libc.so.6
#1  0x7ff523559448 in abort () from /lib/x86_64-linux-gnu/libc.so.6
#2  0x7ff5235961b4 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#3  0x7ff52359b98e in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#4  0x7ff52359c923 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#5  0x7ff523b06c83 in apr_allocator_destroy (allocator=0x7ff4de50)
at memory/unix/apr_pools.c:152
#6  0x556e7986bbef in task_destroy (m=0x7ff46402a028,
task=0x7ff4d0029f00,
called_from_master=0) at h2_mplx.c:400
#7  0x556e7986ce5b in task_done_iter (ctx=,
val=) at h2_mplx.c:1064
#8  0x7ff523afe5e6 in apr_hash_do (
comp=comp@entry=0x556e7987f180 ,
rec=rec@entry=0x7ff5037e5480,
ht=) at tables/apr_hash.c:542
#9  0x556e7987fb5f in h2_ihash_iter (ih=,
fn=fn@entry=0x556e7986ce50 ,
ctx=ctx@entry=0x7ff46402a028)
at h2_util.c:315
#10 0x556e7986d463 in h2_mplx_release_and_join (m=0x7ff46402a028,
wait=0x7ff464029fd0) at h2_mplx.c:619
#11 0x556e79872ae4 in session_pool_cleanup (data=0x7ff464020318)
at h2_session.c:827
#12 0x7ff523b0748e in run_cleanups (cref=0x7ff464029fa8)
at memory/unix/apr_pools.c:2352
#13 apr_pool_destroy (pool=0x7ff464029f38) at memory/unix/apr_pools.c:804
#14 0x7ff523b07745 in apr_pool_clear (pool=0x7ff4fc0150e8)
at memory/unix/apr_pools.c:769
#15 0x556e798aa698 in ap_push_pool (queue_info=0x1e2e,
pool_to_recycle=0x1ebb) at fdqueue.c:234
#16 0x556e798a59da in process_lingering_close (cs=0x7ff4fc015378,
pfd=0x556e7b8bd888) at event.c:1513
#17 0x556e798a9510 in listener_thread (thd=0x1e2e,
dummy=0x547b44ea3e1b3)
at event.c:1837
#18 0x7ff5238d60a4 in start_thread ()
   from /lib/x86_64-linux-gnu/libpthread.so.0
#19 0x7ff52360b62d in clone () from /lib/x86_64-linux-gnu/libc.so.6

and

Program terminated with signal SIGSEGV, Segmentation fault.
#0  apr_bucket_alloc (size=size@entry=64,
list=list@entry=0x3fb50d4a7aeb1d49)
at buckets/apr_buckets_alloc.c:128
#0  apr_bucket_alloc (size=size@entry=64,
list=list@entry=0x3fb50d4a7aeb1d49)
at buckets/apr_buckets_alloc.c:128
#1  0x7ff523d2b1d3 in apr_bucket_heap_create (
buf=0x7ff51003b3a8 "
\311\021A\216y\034\347\034Wy\360\343R\275\226o\020iw\227r\337\377",
length=1300, free_func=0x7ff523d2ab50 ,
list=0x3fb50d4a7aeb1d49) at buckets/apr_buckets_heap.c:81
#2  0x556e79884f85 in append_scratch (io=0x7ff4440377c8)
at h2_conn_io.c:165
#3  0x556e79884ffa in assure_scratch_space (io=0x7ff4440377c8)
at h2_conn_io.c:182
#4  0x556e79885ce8 in h2_conn_io_pass (io=io@entry=0x7ff4440377c8,
bb=0x7ff444133698) at h2_conn_io.c:393
#5  0x556e798732be in on_send_data_cb (ngh2=,
frame=, framehd=, length=1291,
source=, userp=0x7ff444037780) at h2_session.c:648
#6  0x7ff5243dde95 in ?? () from
/usr/lib/x86_64-linux-gnu/libnghttp2.so.14
#7  0x7ff5243deea9 in nghttp2_session_send ()
   from /usr/lib/x86_64-linux-gnu/libnghttp2.so.14
#8  0x556e79875857 in h2_session_send (session=0x7ff444037780)
at h2_session.c:1376
#9  0x556e79878b6c in h2_session_process (session=0x7ff444037780,
async=2062228809) at h2_session.c:2208
#10 0x556e79867788 in h2_conn_run (ctx=0x7ff4440376b0, c=0x7ff51003b6a8)
at h2_conn.c:214
#11 0x556e7986a421 in h2_h2_process_conn (c=0x7ff51003b6a8) at
h2_h2.c:658
#12 0x556e7980d050 in ap_run_process_connection (c=0x7ff51003b6a8)
at connection.c:42
#13 0x556e798a7590 in process_socket (my_thread_num=,
my_child_num=, cs=0x7ff51003b618, sock=,
p=, thd=) at event.c:1134
#14 worker_thread (thd=0x40, dummy=0x3fb50d4a7aeb1d49) at event.c:2137
#15 0x7ff5238d60a4 in start_thread ()
   from /lib/x86_64-linux-gnu/libpthread.so.0
#16 0x7ff52360b62d in clone () from /lib/x86_64-linux-gnu/libc.so.6

Greets,
Stefan

Am 02.02.2017 um 11:09 schrieb Yann Ylavic:
> Hi Stefan,
> 
> On Tue, Jan 31, 2017 at 4:01 PM, Stefan Priebe - Profihost AG
> <s.pri...@profihost.ag> wrote:
>>
>> any ideas?
> 
> I wonder if the attached patch (related to mod_ssl and proposed for
> another segfault report) could help in your case.
> 
> Would you mind give it a try?
> 
> 
> Thanks,
> Yann.
> 

Re: mod_http2 and Frequent wake-ups for mpm_event

[Stefan Priebe - Profihost AG]

Am 02.02.2017 um 11:09 schrieb Yann Ylavic:
> Hi Stefan,
> 
> On Tue, Jan 31, 2017 at 4:01 PM, Stefan Priebe - Profihost AG
> <s.pri...@profihost.ag> wrote:
>>
>> any ideas?
> 
> I wonder if the attached patch (related to mod_ssl and proposed for
> another segfault report) could help in your case.
> 
> Would you mind give it a try?

Up and running. It will take some time until i can report back as those
crashes are very rare.

Stefan

Re: mod_http2 and Frequent wake-ups for mpm_event

[Stefan Priebe - Profihost AG]

Hi Yann,
  Hi Stefan,

any ideas?

Stefan

Am 27.01.2017 um 20:30 schrieb Ruediger Pluem:
> 
> 
> On 01/27/2017 08:10 PM, Stefan Priebe - Profihost AG wrote:
>> Hi Yann,
>>
>> this is now a segfault with mod_http2 + beam v4 patch + mpm v7 patch:
>> Program terminated with signal SIGSEGV, Segmentation fault.
>> #0  allocator_free (node=0x0, allocator=0x7f8974080c90)
>> at memory/unix/apr_pools.c:381
>> #0  allocator_free (node=0x0, allocator=0x7f8974080c90)
>> at memory/unix/apr_pools.c:381
>> #1  apr_pool_clear (pool=0x7f89740830d8) at memory/unix/apr_pools.c:793
> 
> pool from above != pool_to_recycle from below is strange.
> 
>> #2  0x5611421447a8 in ap_push_pool (queue_info=0x0,
> 
> queue_info NULL? That looks bad.
> 
>> pool_to_recycle=0x7f8974080c98) at fdqueue.c:234
>> #3  0x56114213faea in process_lingering_close (cs=0x7f8974083368,
>> pfd=0x561142cc97f8) at event.c:1513
>> #4  0x561142143620 in listener_thread (thd=0x0, dummy=0x54716c3684bbc)
>> at event.c:1837
>> #5  0x7f8989bb20a4 in start_thread ()
>>from /lib/x86_64-linux-gnu/libpthread.so.0
>> #6  0x7f89898e762d in clone () from /lib/x86_64-linux-gnu/libc.so.6
> 
> Regards
> 
> Rüdiger
> 

Re: mod_http2 and canceled requests logs status 500 internal server error code

[Stefan Priebe - Profihost AG]

Hi,

Am 27.01.2017 um 18:59 schrieb Stefan Eissing:
> Out now: https://github.com/icing/mod_h2/releases/tag/1.8.11 fresh from 
> Apache svn trunk+2.4.x
> 
> I could not reproduce the 500 behaviour, but added a check on the recent 
> change on response generation. Hope this helps.

thanks will report back most probably on monday.

Stefan

> Cheers, Stefan
> 
>> Am 26.01.2017 um 16:36 schrieb Stefan Eissing <stefan.eiss...@greenbytes.de>:
>>
>> Hi, will do that tomorrow.
>>
>>> Am 26.01.2017 um 16:35 schrieb Stefan Priebe - Profihost AG 
>>> <s.pri...@profihost.ag>:
>>>
>>> Hi Stefan,
>>>
>>> did you already had the time to look at the 500 status code in case of
>>> canceled requests?
>>>
>>> Stefan
>>>
>>> Am 25.01.2017 um 15:55 schrieb Stefan Priebe - Profihost AG:
>>>>
>>>> Am 25.01.2017 um 15:50 schrieb Stefan Eissing:
>>>>>
>>>>>> Am 25.01.2017 um 15:31 schrieb Stefan Priebe - Profihost AG 
>>>>>> <s.pri...@profihost.ag>:
>>>>>>
>>>>>> Chrome docs says:
>>>>>>
>>>>>> "cancelled / net::ERR_ABORTED is intended to only be generated when a
>>>>>> user action causes a load to be interrupted. This can happen when a new
>>>>>> navigation interrupts an existing one, or when the user clicks the STOP
>>>>>> button"
>>>>>>
>>>>>> This matches my click assumption. I think it's just wrong that mod_http2
>>>>>> generates a 500 in the logs.
>>>>>
>>>>> I agree. Might be a new one related to the 304 fix in 1.8.10.
>>>>
>>>> Would be great if you can have a look. It confuses the users while
>>>> reading their logs.
>>>>
>>>> I also saw some more entries with 0 byte size in logs if there is no
>>>> body. http/1.1 logs the header size as well.
>>>>
>>>> Greets,
>>>> Stefan
>>>>
>>>>>>
>>>>>> Stefan
>>>>>>
>>>>>> Am 25.01.2017 um 15:27 schrieb Stefan Priebe - Profihost AG:
>>>>>>> Am 25.01.2017 um 14:57 schrieb Yann Ylavic:
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> On Wed, Jan 25, 2017 at 2:46 PM, Stefan Priebe - Profihost AG
>>>>>>>> <s.pri...@profihost.ag> wrote:
>>>>>>>>>
>>>>>>>>> Is this a bug or a feature? Is it correct that it logs a 500 code?
>>>>>>>>
>>>>>>>> ErrorLog's file should tell more about the error.
>>>>>>>
>>>>>>> No error log does not have an entry with the same or nearly the same
>>>>>>> timestamp.
>>>>>>>
>>>>>>> Stefan
>>>>>>>
>>>>>>>>
>>>>>>>> Regards,
>>>>>>>> Yann.
>>>>>>>>
>>>>>
>>>>> Stefan Eissing
>>>>>
>>>>> bytes GmbH
>>>>> Hafenstrasse 16
>>>>> 48155 Münster
>>>>> www.greenbytes.de
>>>>>
>>
>> Stefan Eissing
>>
>> bytes GmbH
>> Hafenstrasse 16
>> 48155 Münster
>> www.greenbytes.de
>>
> 
> Stefan Eissing
> 
> bytes GmbH
> Hafenstrasse 16
> 48155 Münster
> www.greenbytes.de
> 

Re: mod_http2 and Frequent wake-ups for mpm_event

[Stefan Priebe - Profihost AG]

Hi Yann,

this is now a segfault with mod_http2 + beam v4 patch + mpm v7 patch:
Program terminated with signal SIGSEGV, Segmentation fault.
#0  allocator_free (node=0x0, allocator=0x7f8974080c90)
at memory/unix/apr_pools.c:381
#0  allocator_free (node=0x0, allocator=0x7f8974080c90)
at memory/unix/apr_pools.c:381
#1  apr_pool_clear (pool=0x7f89740830d8) at memory/unix/apr_pools.c:793
#2  0x5611421447a8 in ap_push_pool (queue_info=0x0,
pool_to_recycle=0x7f8974080c98) at fdqueue.c:234
#3  0x56114213faea in process_lingering_close (cs=0x7f8974083368,
pfd=0x561142cc97f8) at event.c:1513
#4  0x561142143620 in listener_thread (thd=0x0, dummy=0x54716c3684bbc)
at event.c:1837
#5  0x7f8989bb20a4 in start_thread ()
   from /lib/x86_64-linux-gnu/libpthread.so.0
#6  0x7f89898e762d in clone () from /lib/x86_64-linux-gnu/libc.so.6


Stefan
Am 25.01.2017 um 12:10 schrieb Stefan Priebe - Profihost AG:
> Am 25.01.2017 um 10:46 schrieb Stefan Eissing:
>> Thanks Yann!
>>
>> Stefan: here is the patch as committed to trunk:
> 
> Up and running. Will report back.
> 
>>
>>
>> Cheers, Stefan
>>
>>> Am 25.01.2017 um 01:41 schrieb Yann Ylavic <ylavic@gmail.com>:
>>>
>>> Hi Stefan,
>>>
>>> On Tue, Jan 24, 2017 at 1:37 PM, Stefan Eissing
>>> <stefan.eiss...@greenbytes.de> wrote:
>>>> Yann, thanks for the patch. I agree that the cleanups need to be killed in 
>>>> the right place. Not certain if it was wrong before, but that part is not 
>>>> easy to see for every combination.
>>>>
>>>> I did some rework and hope this makes it more readable. If you find the 
>>>> time to look at it, feedback welcome.
>>>
>>> I still fear that if beam->pool gets destroyed while both
>>> beam_send_cleanup() and beam_cleanup() are registered, the former is
>>> called twice.
>>>
>>> I'd change:
>>>if (safe_send) {
>>>if (beam->send_pool && beam->send_pool != beam->pool) {
>>>apr_pool_cleanup_kill(beam->send_pool, beam, beam_send_cleanup);
>>>}
>>>status = beam_send_cleanup(beam);
>>>}
>>>
>>> with:
>>>if (safe_send) {
>>>if (beam->send_pool) {
>>>if (beam->send_pool != beam->pool) {
>>>apr_pool_cleanup_kill(beam->send_pool, beam, 
>>> beam_send_cleanup);
>>>}
>>>status = beam_send_cleanup(beam);
>>>}
>>>}
>>>
>>> since in the above case beam_send_cleanup is run first and sets 
>>> send_pool=NULL.
>>>
>>> Attached v3 with this only change w.r.t. v2.
>>> Otherwise, looks good to me, thanks!
>>> 
>>
>> Stefan Eissing
>>
>> bytes GmbH
>> Hafenstrasse 16
>> 48155 Münster
>> www.greenbytes.de
>>

Re: mod_http2 and canceled requests logs status 500 internal server error code

[Stefan Priebe - Profihost AG]

Hi Stefan,

did you already had the time to look at the 500 status code in case of
canceled requests?

Stefan

Am 25.01.2017 um 15:55 schrieb Stefan Priebe - Profihost AG:
> 
> Am 25.01.2017 um 15:50 schrieb Stefan Eissing:
>>
>>> Am 25.01.2017 um 15:31 schrieb Stefan Priebe - Profihost AG 
>>> <s.pri...@profihost.ag>:
>>>
>>> Chrome docs says:
>>>
>>> "cancelled / net::ERR_ABORTED is intended to only be generated when a
>>> user action causes a load to be interrupted. This can happen when a new
>>> navigation interrupts an existing one, or when the user clicks the STOP
>>> button"
>>>
>>> This matches my click assumption. I think it's just wrong that mod_http2
>>> generates a 500 in the logs.
>>
>> I agree. Might be a new one related to the 304 fix in 1.8.10.
> 
> Would be great if you can have a look. It confuses the users while
> reading their logs.
> 
> I also saw some more entries with 0 byte size in logs if there is no
> body. http/1.1 logs the header size as well.
> 
> Greets,
> Stefan
> 
>>>
>>> Stefan
>>>
>>> Am 25.01.2017 um 15:27 schrieb Stefan Priebe - Profihost AG:
>>>> Am 25.01.2017 um 14:57 schrieb Yann Ylavic:
>>>>> Hi,
>>>>>
>>>>> On Wed, Jan 25, 2017 at 2:46 PM, Stefan Priebe - Profihost AG
>>>>> <s.pri...@profihost.ag> wrote:
>>>>>>
>>>>>> Is this a bug or a feature? Is it correct that it logs a 500 code?
>>>>>
>>>>> ErrorLog's file should tell more about the error.
>>>>
>>>> No error log does not have an entry with the same or nearly the same
>>>> timestamp.
>>>>
>>>> Stefan
>>>>
>>>>>
>>>>> Regards,
>>>>> Yann.
>>>>>
>>
>> Stefan Eissing
>>
>> bytes GmbH
>> Hafenstrasse 16
>> 48155 Münster
>> www.greenbytes.de
>>

Re: mod_http2 and canceled requests logs status 500 internal server error code

[Stefan Priebe - Profihost AG]

Hi,

Am 25.01.2017 um 15:59 schrieb Eric Covener:
> On Wed, Jan 25, 2017 at 9:55 AM, Stefan Priebe - Profihost AG
> <s.pri...@profihost.ag> wrote:
>> I also saw some more entries with 0 byte size in logs if there is no
>> body. http/1.1 logs the header size as well.
> 
> What's your logformat?  Apache doesn't count headers in %b.  If it's
> zero bytes, a - is logged.  I suggest adding %X while looking at this
> issue to.

i'm using and we're calculating some value from them. Need to recheck.
Main problem can't reproduce this with my "slow" wlan notebook connection.
%O %{instream}n %I

Stefan

Re: mod_http2 and canceled requests logs status 500 internal server error code

[Stefan Priebe - Profihost AG]

Am 25.01.2017 um 15:50 schrieb Stefan Eissing:
> 
>> Am 25.01.2017 um 15:31 schrieb Stefan Priebe - Profihost AG 
>> <s.pri...@profihost.ag>:
>>
>> Chrome docs says:
>>
>> "cancelled / net::ERR_ABORTED is intended to only be generated when a
>> user action causes a load to be interrupted. This can happen when a new
>> navigation interrupts an existing one, or when the user clicks the STOP
>> button"
>>
>> This matches my click assumption. I think it's just wrong that mod_http2
>> generates a 500 in the logs.
> 
> I agree. Might be a new one related to the 304 fix in 1.8.10.

Would be great if you can have a look. It confuses the users while
reading their logs.

I also saw some more entries with 0 byte size in logs if there is no
body. http/1.1 logs the header size as well.

Greets,
Stefan

>>
>> Stefan
>>
>> Am 25.01.2017 um 15:27 schrieb Stefan Priebe - Profihost AG:
>>> Am 25.01.2017 um 14:57 schrieb Yann Ylavic:
>>>> Hi,
>>>>
>>>> On Wed, Jan 25, 2017 at 2:46 PM, Stefan Priebe - Profihost AG
>>>> <s.pri...@profihost.ag> wrote:
>>>>>
>>>>> Is this a bug or a feature? Is it correct that it logs a 500 code?
>>>>
>>>> ErrorLog's file should tell more about the error.
>>>
>>> No error log does not have an entry with the same or nearly the same
>>> timestamp.
>>>
>>> Stefan
>>>
>>>>
>>>> Regards,
>>>> Yann.
>>>>
> 
> Stefan Eissing
> 
> bytes GmbH
> Hafenstrasse 16
> 48155 Münster
> www.greenbytes.de
> 

Re: mod_http2 and canceled requests logs status 500 internal server error code

[Stefan Priebe - Profihost AG]

Chrome docs says:

"cancelled / net::ERR_ABORTED is intended to only be generated when a
user action causes a load to be interrupted. This can happen when a new
navigation interrupts an existing one, or when the user clicks the STOP
button"

This matches my click assumption. I think it's just wrong that mod_http2
generates a 500 in the logs.

Stefan

Am 25.01.2017 um 15:27 schrieb Stefan Priebe - Profihost AG:
> Am 25.01.2017 um 14:57 schrieb Yann Ylavic:
>> Hi,
>>
>> On Wed, Jan 25, 2017 at 2:46 PM, Stefan Priebe - Profihost AG
>> <s.pri...@profihost.ag> wrote:
>>>
>>> Is this a bug or a feature? Is it correct that it logs a 500 code?
>>
>> ErrorLog's file should tell more about the error.
> 
> No error log does not have an entry with the same or nearly the same
> timestamp.
> 
> Stefan
> 
>>
>> Regards,
>> Yann.
>>

Re: mod_http2 and canceled requests logs status 500 internal server error code

[Stefan Priebe - Profihost AG]

Am 25.01.2017 um 14:57 schrieb Yann Ylavic:
> Hi,
> 
> On Wed, Jan 25, 2017 at 2:46 PM, Stefan Priebe - Profihost AG
> <s.pri...@profihost.ag> wrote:
>>
>> Is this a bug or a feature? Is it correct that it logs a 500 code?
> 
> ErrorLog's file should tell more about the error.

No error log does not have an entry with the same or nearly the same
timestamp.

Stefan

> 
> Regards,
> Yann.
> 

mod_http2 and canceled requests logs status 500 internal server error code

[Stefan Priebe - Profihost AG]

Hello,

while digging through some apache logfiles i saw some strange entries:
"GET URL HTTP/2.0" 500 0

They all had a 500 status error code and a size of 0.

I'm able to reproduce them while clicking fast on a php site. For each
log line with status code 500 i get a STATUS (canceled) in my chrome
developer console. It seems it happens if i click a new link before the
first click got a response back from the server.

Is this a bug or a feature? Is it correct that it logs a 500 code?

Greets,
Stefan

Re: mod_http2 and Frequent wake-ups for mpm_event

[Stefan Priebe - Profihost AG]

Am 25.01.2017 um 10:46 schrieb Stefan Eissing:
> Thanks Yann!
> 
> Stefan: here is the patch as committed to trunk:

Up and running. Will report back.

> 
> 
> Cheers, Stefan
> 
>> Am 25.01.2017 um 01:41 schrieb Yann Ylavic :
>>
>> Hi Stefan,
>>
>> On Tue, Jan 24, 2017 at 1:37 PM, Stefan Eissing
>>  wrote:
>>> Yann, thanks for the patch. I agree that the cleanups need to be killed in 
>>> the right place. Not certain if it was wrong before, but that part is not 
>>> easy to see for every combination.
>>>
>>> I did some rework and hope this makes it more readable. If you find the 
>>> time to look at it, feedback welcome.
>>
>> I still fear that if beam->pool gets destroyed while both
>> beam_send_cleanup() and beam_cleanup() are registered, the former is
>> called twice.
>>
>> I'd change:
>>if (safe_send) {
>>if (beam->send_pool && beam->send_pool != beam->pool) {
>>apr_pool_cleanup_kill(beam->send_pool, beam, beam_send_cleanup);
>>}
>>status = beam_send_cleanup(beam);
>>}
>>
>> with:
>>if (safe_send) {
>>if (beam->send_pool) {
>>if (beam->send_pool != beam->pool) {
>>apr_pool_cleanup_kill(beam->send_pool, beam, 
>> beam_send_cleanup);
>>}
>>status = beam_send_cleanup(beam);
>>}
>>}
>>
>> since in the above case beam_send_cleanup is run first and sets 
>> send_pool=NULL.
>>
>> Attached v3 with this only change w.r.t. v2.
>> Otherwise, looks good to me, thanks!
>> 
> 
> Stefan Eissing
> 
> bytes GmbH
> Hafenstrasse 16
> 48155 Münster
> www.greenbytes.de
> 

Re: mod_http2 and Frequent wake-ups for mpm_event

[Stefan Priebe - Profihost AG]

Hi Yann,

Am 25.01.2017 um 01:41 schrieb Yann Ylavic:
> Hi Stefan,
> 
> On Tue, Jan 24, 2017 at 1:37 PM, Stefan Eissing
>  wrote:
>> Yann, thanks for the patch. I agree that the cleanups need to be killed in 
>> the right place. Not certain if it was wrong before, but that part is not 
>> easy to see for every combination.
>>
>> I did some rework and hope this makes it more readable. If you find the time 
>> to look at it, feedback welcome.
> 
> I still fear that if beam->pool gets destroyed while both
> beam_send_cleanup() and beam_cleanup() are registered, the former is
> called twice.
> 
> I'd change:
> if (safe_send) {
> if (beam->send_pool && beam->send_pool != beam->pool) {
> apr_pool_cleanup_kill(beam->send_pool, beam, beam_send_cleanup);
> }
> status = beam_send_cleanup(beam);
> }
> 
> with:
> if (safe_send) {
> if (beam->send_pool) {
> if (beam->send_pool != beam->pool) {
> apr_pool_cleanup_kill(beam->send_pool, beam, 
> beam_send_cleanup);
> }
> status = beam_send_cleanup(beam);
> }
> }
> 
> since in the above case beam_send_cleanup is run first and sets 
> send_pool=NULL.
> 
> Attached v3 with this only change w.r.t. v2.
> Otherwise, looks good to me, thanks!

this one also contains changes to the even mpm, mod_proxy and
mod_slotmem_shm. Is this expected?

Stefan
> 

Re: mod_http2 and Frequent wake-ups for mpm_event

[Stefan Priebe - Profihost AG]

Am 24.01.2017 um 14:22 schrieb Yann Ylavic:
> On Tue, Jan 24, 2017 at 10:02 AM, Stefan Priebe - Profihost AG
> <s.pri...@profihost.ag> wrote:
>>
>> Am 23.01.2017 um 23:42 schrieb Yann Ylavic:
>>> On Mon, Jan 23, 2017 at 11:37 PM, Yann Ylavic <ylavic@gmail.com> wrote:
>>>> Hi Stefan,
>>>>
>>>> On Mon, Jan 23, 2017 at 9:54 PM, Stefan Eissing
>>>> <stefan.eiss...@greenbytes.de> wrote:
>>>>>
>>>>> I am not aware of any special expectations now. Almost all is triggered 
>>>>> by (parent) pool cleanups and is therefore more deterministic than 
>>>>> before. The only explicit destroy is done on finished streams and slave 
>>>>> connections no longer used. When the master conn disappears, all is 
>>>>> deallocated as the force wills it.
>>>>
>>>> I wonder if the attached patch makes sense.
>>>> If beam_{recv,send}_cleanup() are to be executed on (parent) pool
>>>> destroy, there will be before beam_cleanup() itelf (which also calls
>>>> beam_send_cleanup() explicitly), so it should avoid potential a double
>>>> cleanup in this case.
>>>>
>>>> WDYT?
>>>
>>> Please ignore the last (garbage) hunk.
>>
>> last garbage hunk?
>>
>> This one?
> 
> Yes, this change is not necessary/suitable.

Thanks. Would currently wait if you had a look at stefans patch. Or
isn't that the same but "better"?

Stefan

Re: mod_http2 and Frequent wake-ups for mpm_event

[Stefan Priebe - Profihost AG]

Am 23.01.2017 um 23:42 schrieb Yann Ylavic:
> On Mon, Jan 23, 2017 at 11:37 PM, Yann Ylavic  wrote:
>> Hi Stefan,
>>
>> On Mon, Jan 23, 2017 at 9:54 PM, Stefan Eissing
>>  wrote:
>>>
>>> I am not aware of any special expectations now. Almost all is triggered by 
>>> (parent) pool cleanups and is therefore more deterministic than before. The 
>>> only explicit destroy is done on finished streams and slave connections no 
>>> longer used. When the master conn disappears, all is deallocated as the 
>>> force wills it.
>>
>> I wonder if the attached patch makes sense.
>> If beam_{recv,send}_cleanup() are to be executed on (parent) pool
>> destroy, there will be before beam_cleanup() itelf (which also calls
>> beam_send_cleanup() explicitly), so it should avoid potential a double
>> cleanup in this case.
>>
>> WDYT?
> 
> Please ignore the last (garbage) hunk.

last garbage hunk?

This one?
@@ -520,8 +529,8 @@ static apr_status_t beam_cleanup(void *data)
  * beam should have lost its mutex protection, meaning
  * it is no longer used multi-threaded and we can safely
  * purge all remaining sender buckets. */
-apr_pool_cleanup_kill(beam->send_pool, beam,
beam_send_cleanup);
 ap_assert(!beam->m_enter);
+beam_set_send_pool(beam, NULL);
 beam_send_cleanup(beam);
 }
 ap_assert(H2_BPROXY_LIST_EMPTY(>proxies));


Stefan

Re: mod_http2 and Frequent wake-ups for mpm_event

[Stefan Priebe - Profihost AG]

Am 23.01.2017 um 21:54 schrieb Stefan Eissing:
> 
>> Am 22.01.2017 um 22:22 schrieb Yann Ylavic :
>>
>> @icing: Any special expectation in mod_h2 with regard to mpm workers
>> threads' lifetime (or keepalive connections that should stay alive for
>> the configured limit)?
>> I see that beam buckets make use of thread local storage/keys for
>> locking, and that they also handle the double cleanup like eoc buckets
>> did before 1.8.9, but can't follow all the paths yet.
>> Maybe something to look at there?
> 
> - With your help, mod_http2 1.8.9 simplified cleanup by triggering it by the 
> connection pool clean up only. That helped. The bucket beam still register 
> for pool cleanup which could be done without, but I think it's a good 
> failsafe.
> - The thread local is used for recursive locking and, once the outermost lock 
> is released, are supposed to be NULL again. This I would like to eliminate 
> one day.
> - the special handling for apr_files is gone as well. That was a work-around 
> when shared file buckets were transported through a bucket beam. No more. 
> Shared file buckets are copied now. Only file buckets with refcount == 1 are 
> beamed now. That makes the beam the sole controller of the lifetime and makes 
> setasides on the master connection work without special handling.
> 
> I am not aware of any special expectations now. Almost all is triggered by 
> (parent) pool cleanups and is therefore more deterministic than before. The 
> only explicit destroy is done on finished streams and slave connections no 
> longer used. When the master conn disappears, all is deallocated as the force 
> wills it.

Thanks and + 1 - only two crashes in 48h. ,-) before i had around 2 per
hour. Need some time to debug them. Don't have dumps yet.

Is there an easier way to get core dumps than manually starting httpd
while using systemd?
ulimit -c unlimited
httpd -k start

Greets,
Stefan

> 
> Stefan Eissing
> 
> bytes GmbH
> Hafenstrasse 16
> 48155 Münster
> www.greenbytes.de
>