Re: mod_http2 and Frequent wake-ups for mpm_event

[Stefan Priebe - Profihost AG]

Am 19.01.2017 um 08:26 schrieb Stefan Eissing:
> Will look into this. With which patch do you have it most frequently?

It happens pretty fast while using:
Apache 2.4.25  + mpm_event V6 + mod_http2 V1.8.8

Greets,
Stefan

> Cheers, Stefan
> 
>> Am 19.01.2017 um 08:22 schrieb Stefan Priebe - Profihost AG 
>> :
>>
>> Hi Stefan,
>>
>> Apache 2.4.25 + mpm_event V7 and core mod_http2:
>> Core was generated by `/usr/local/apache2/bin/httpd -k start'.
>> Program terminated with signal SIGSEGV, Segmentation fault.
>> #0  0x00521d98 in h2_stream_out_prepare ()
>> (gdb) bt
>> #0  0x00521d98 in h2_stream_out_prepare ()
>> #1  0x7f54f27eba80 in ?? ()
>> #2  0x7f54f27eba8c in ?? ()
>> #3  0x7f54f27eba90 in ?? ()
>> #4  0x7f54ef00e0a0 in ?? ()
>> #5  0x7f54ef00e9a8 in ?? ()
>> #6  0x7f54f27ebac0 in ?? ()
>> #7  0x in ?? ()
>>
>> so it's crashing with mpm_event and without mpm_event. And it does with
>> core mod_http2 and mod_http2 v1.8.8.
>>
>> Is a regression in mod_http2 possible?
>>
>> Greets,
>> Stefan
>>> Am 19.01.2017 um 07:55 schrieb Stefan Priebe - Profihost AG:
>>> Dear Stefan,
>>> dear Yann,
>>>
>>> a longer test run shows it's also crashing without any mpm_event patch
>>> at all. So I'm really sorry. It just starts crashing faster with the
>>> mpm_event patches.
>>>
>>> I'll now retest regarding mod_http2 core, plain apache 2.4.25 and
>>> mod_http v1.8.8
>>>
>>> may be it's a regression in 2.4.25 itself or in mod_http2.
>>>
>>> Stefan
>>>
 Am 19.01.2017 um 00:52 schrieb Yann Ylavic:
 On Wed, Jan 18, 2017 at 10:49 PM, Stefan Priebe - Profihost AG
  wrote:
>
> v5 does not apply to 2.4.25. If you can send me a v5 version that
> applies to 2.4.25 i'll test.

 Here it is (slightly modified to fix possible deadlocks, per r1762742
 and r1774538).

 Thanks!

> 

Re: mod_http2 and Frequent wake-ups for mpm_event

[Stefan Eissing]

Will look into this. With which patch do you have it most frequently?

Cheers, Stefan

> Am 19.01.2017 um 08:22 schrieb Stefan Priebe - Profihost AG 
> :
> 
> Hi Stefan,
> 
> Apache 2.4.25 + mpm_event V7 and core mod_http2:
> Core was generated by `/usr/local/apache2/bin/httpd -k start'.
> Program terminated with signal SIGSEGV, Segmentation fault.
> #0  0x00521d98 in h2_stream_out_prepare ()
> (gdb) bt
> #0  0x00521d98 in h2_stream_out_prepare ()
> #1  0x7f54f27eba80 in ?? ()
> #2  0x7f54f27eba8c in ?? ()
> #3  0x7f54f27eba90 in ?? ()
> #4  0x7f54ef00e0a0 in ?? ()
> #5  0x7f54ef00e9a8 in ?? ()
> #6  0x7f54f27ebac0 in ?? ()
> #7  0x in ?? ()
> 
> so it's crashing with mpm_event and without mpm_event. And it does with
> core mod_http2 and mod_http2 v1.8.8.
> 
> Is a regression in mod_http2 possible?
> 
> Greets,
> Stefan
>> Am 19.01.2017 um 07:55 schrieb Stefan Priebe - Profihost AG:
>> Dear Stefan,
>> dear Yann,
>> 
>> a longer test run shows it's also crashing without any mpm_event patch
>> at all. So I'm really sorry. It just starts crashing faster with the
>> mpm_event patches.
>> 
>> I'll now retest regarding mod_http2 core, plain apache 2.4.25 and
>> mod_http v1.8.8
>> 
>> may be it's a regression in 2.4.25 itself or in mod_http2.
>> 
>> Stefan
>> 
>>> Am 19.01.2017 um 00:52 schrieb Yann Ylavic:
>>> On Wed, Jan 18, 2017 at 10:49 PM, Stefan Priebe - Profihost AG
>>>  wrote:
 
 v5 does not apply to 2.4.25. If you can send me a v5 version that
 applies to 2.4.25 i'll test.
>>> 
>>> Here it is (slightly modified to fix possible deadlocks, per r1762742
>>> and r1774538).
>>> 
>>> Thanks!
>>> 

Re: mod_http2 and Frequent wake-ups for mpm_event

[Stefan Priebe - Profihost AG]

Hi Stefan,

Apache 2.4.25 + mpm_event V7 and core mod_http2:
Core was generated by `/usr/local/apache2/bin/httpd -k start'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x00521d98 in h2_stream_out_prepare ()
(gdb) bt
#0  0x00521d98 in h2_stream_out_prepare ()
#1  0x7f54f27eba80 in ?? ()
#2  0x7f54f27eba8c in ?? ()
#3  0x7f54f27eba90 in ?? ()
#4  0x7f54ef00e0a0 in ?? ()
#5  0x7f54ef00e9a8 in ?? ()
#6  0x7f54f27ebac0 in ?? ()
#7  0x in ?? ()

so it's crashing with mpm_event and without mpm_event. And it does with
core mod_http2 and mod_http2 v1.8.8.

Is a regression in mod_http2 possible?

Greets,
Stefan
Am 19.01.2017 um 07:55 schrieb Stefan Priebe - Profihost AG:
> Dear Stefan,
>  dear Yann,
> 
> a longer test run shows it's also crashing without any mpm_event patch
> at all. So I'm really sorry. It just starts crashing faster with the
> mpm_event patches.
> 
> I'll now retest regarding mod_http2 core, plain apache 2.4.25 and
> mod_http v1.8.8
> 
> may be it's a regression in 2.4.25 itself or in mod_http2.
> 
> Stefan
> 
> Am 19.01.2017 um 00:52 schrieb Yann Ylavic:
>> On Wed, Jan 18, 2017 at 10:49 PM, Stefan Priebe - Profihost AG
>>  wrote:
>>>
>>> v5 does not apply to 2.4.25. If you can send me a v5 version that
>>> applies to 2.4.25 i'll test.
>>
>> Here it is (slightly modified to fix possible deadlocks, per r1762742
>> and r1774538).
>>
>> Thanks!
>>

Re: [proposed] 2.4 Maintenance SIG

[Stefan Eissing]

> Am 19.01.2017 um 06:34 schrieb William A Rowe Jr :
> 
>> On Wed, Jan 18, 2017 at 7:35 PM, Eric Covener  wrote:
>>> On Wed, Jan 18, 2017 at 6:12 PM, William A Rowe Jr  
>>> wrote:
>>> I'm wondering if there is anyone interested in a regression-fix-only 2.4.26 
>>> that
>>> finally proves to be a workable upgrade for all httpd users?
>> 
>> It sounds reasonable to me, but I think it's a bit of an oversell --
>> It's just going to be a little bit of stabilization.
>> 
>> AFAICT so far there is:
>> 
>> One 2.4.25 regression committed:
>> 
>>  *) mod_proxy_{ajp,fcgi}: Fix a possible crash when reusing an established
>>backend connection, happening with LogLevel trace2 or higher configured,
>>or at any log level with compilers not detected as C99 compliant (e.g.
>>MSVC on Windows).  [Yann Ylavic]
>> 
>> One older regression listed as a showstopper:
>> 
>> *) PR 60576: 2.4.21 broke PHP-FPM with the patch to strip the bogus 
>> "proxy://"
>>prefix from SCRIPT_FILENAME. We need to revert to the previous behavior
>>ASAP to avoid any further hurdles with FCGI implementations while we 
>> figure
>>this out.
>> 
>> And a fix for an old bug that missed being backported until a bugzilla 
>> review:
>> 
>>  *) mod_proxy_fcgi, mod_fcgid: Fix crashes in ap_fcgi_encoded_env_len() when
>>modules add empty environment variables to the request. PR60275.
>>[]
>> 
>> Is there anything else we should list as a showstopper?
> 
> That is my underlying question; what else qualifies?
> 
> Win32 build fix of mod_status is already in 2.4.x branch two hours following
> the 2.4.25 re-tag, so that also is resolved.
> 
> Yann's proposal to accept the newly-prohibited obs-fold that we approved into
> 2.2.32 would also seem to qualify.
> 
> So far, I haven't heard from an httpd committer who is actually interested in
> our shipping such a release. I'd be happy to tag and roll, but this is more so
> a poll whether there is interest in having non-enhancement 2.4.x releases
> by the PMC/committer core, or whether there is a desire for a  fork for users
> who are not PMC members but are actively interested in pursuing a "stable"
> build.

It will surprise no one that I like to release more often. OTOH I do not like 
to break things.

The current release model clearly does not work well, in my limited experience 
over that last 15 months. Why?

httpd is a rich product used in literally millions of different setups. This 
complexity is beyond our resources to test. Adding new functionality at the 
same time is just too much.

That is why I am sticking to the separate releases of mod_http2 on github. 
Those get new code tested very frequently by adventourous users, finding bugs 
and giving suggestions. The http/2 implementation wouldn't be that far without 
them. 

Distros seem to have realized the problem long ago and make their own httpd 
versions. First time I realized my "httpd 2.4.7" is not the 2.4.7 release was a 
WTF moment.

Personally, I like even/odd version schemes for stable/dev versions. But 
whatever works is fine. There are a lot of successful projects in and outside 
ASF with a good release scheme. Let's steal with pride one we think is good.

Cheers, 

Stefan

Re: mod_http2 and Frequent wake-ups for mpm_event

[Stefan Priebe - Profihost AG]

Dear Stefan,
 dear Yann,

a longer test run shows it's also crashing without any mpm_event patch
at all. So I'm really sorry. It just starts crashing faster with the
mpm_event patches.

I'll now retest regarding mod_http2 core, plain apache 2.4.25 and
mod_http v1.8.8

may be it's a regression in 2.4.25 itself or in mod_http2.

Stefan

Am 19.01.2017 um 00:52 schrieb Yann Ylavic:
> On Wed, Jan 18, 2017 at 10:49 PM, Stefan Priebe - Profihost AG
>  wrote:
>>
>> v5 does not apply to 2.4.25. If you can send me a v5 version that
>> applies to 2.4.25 i'll test.
> 
> Here it is (slightly modified to fix possible deadlocks, per r1762742
> and r1774538).
> 
> Thanks!
> 

Re: [proposed] 2.4 Maintenance SIG

[William A Rowe Jr]

On Wed, Jan 18, 2017 at 7:35 PM, Eric Covener  wrote:
> On Wed, Jan 18, 2017 at 6:12 PM, William A Rowe Jr  
> wrote:
>> I'm wondering if there is anyone interested in a regression-fix-only 2.4.26 
>> that
>> finally proves to be a workable upgrade for all httpd users?
>
> It sounds reasonable to me, but I think it's a bit of an oversell --
> It's just going to be a little bit of stabilization.
>
> AFAICT so far there is:
>
> One 2.4.25 regression committed:
>
>   *) mod_proxy_{ajp,fcgi}: Fix a possible crash when reusing an established
> backend connection, happening with LogLevel trace2 or higher configured,
> or at any log level with compilers not detected as C99 compliant (e.g.
> MSVC on Windows).  [Yann Ylavic]
>
> One older regression listed as a showstopper:
>
>  *) PR 60576: 2.4.21 broke PHP-FPM with the patch to strip the bogus 
> "proxy://"
> prefix from SCRIPT_FILENAME. We need to revert to the previous behavior
> ASAP to avoid any further hurdles with FCGI implementations while we 
> figure
> this out.
>
> And a fix for an old bug that missed being backported until a bugzilla review:
>
>   *) mod_proxy_fcgi, mod_fcgid: Fix crashes in ap_fcgi_encoded_env_len() when
> modules add empty environment variables to the request. PR60275.
> []
>
> Is there anything else we should list as a showstopper?

That is my underlying question; what else qualifies?

Win32 build fix of mod_status is already in 2.4.x branch two hours following
the 2.4.25 re-tag, so that also is resolved.

Yann's proposal to accept the newly-prohibited obs-fold that we approved into
2.2.32 would also seem to qualify.

So far, I haven't heard from an httpd committer who is actually interested in
our shipping such a release. I'd be happy to tag and roll, but this is more so
a poll whether there is interest in having non-enhancement 2.4.x releases
by the PMC/committer core, or whether there is a desire for a  fork for users
who are not PMC members but are actively interested in pursuing a "stable"
build.

Cheers,

Bill

Re: [proposed] 2.4 Maintenance SIG

[Eric Covener]

On Wed, Jan 18, 2017 at 6:12 PM, William A Rowe Jr  wrote:
> I'm wondering if there is anyone interested in a regression-fix-only 2.4.26 
> that
> finally proves to be a workable upgrade for all httpd users?

It sounds reasonable to me, but I think it's a bit of an oversell --
It's just going to be a little bit of stabilization.

AFAICT so far there is:

One 2.4.25 regression committed:

  *) mod_proxy_{ajp,fcgi}: Fix a possible crash when reusing an established
backend connection, happening with LogLevel trace2 or higher configured,
or at any log level with compilers not detected as C99 compliant (e.g.
MSVC on Windows).  [Yann Ylavic]

One older regression listed as a showstopper:

 *) PR 60576: 2.4.21 broke PHP-FPM with the patch to strip the bogus "proxy://"
prefix from SCRIPT_FILENAME. We need to revert to the previous behavior
ASAP to avoid any further hurdles with FCGI implementations while we figure
this out.

And a fix for an old bug that missed being backported until a bugzilla review:

  *) mod_proxy_fcgi, mod_fcgid: Fix crashes in ap_fcgi_encoded_env_len() when
modules add empty environment variables to the request. PR60275.
[]

Is there anything else we should list as a showstopper?

Re: mod_http2 and Frequent wake-ups for mpm_event

[Yann Ylavic]

On Wed, Jan 18, 2017 at 10:49 PM, Stefan Priebe - Profihost AG
 wrote:
>
> v5 does not apply to 2.4.25. If you can send me a v5 version that
> applies to 2.4.25 i'll test.

Here it is (slightly modified to fix possible deadlocks, per r1762742
and r1774538).

Thanks!
Index: server/mpm/event/event.c
===
--- server/mpm/event/event.c	(revision 1778313)
+++ server/mpm/event/event.c	(working copy)
@@ -100,6 +100,8 @@
 #include  /* for INT_MAX */
 
 
+#define VOLATILE_READ(T, x) (*(volatile T *)&(x))
+
 /* Limit on the total --- clients will be locked out if more servers than
  * this are needed.  It is intended solely to keep the server from crashing
  * when things get out of hand.
@@ -177,6 +179,7 @@ static int dying = 0;
 static int workers_may_exit = 0;
 static int start_thread_may_exit = 0;
 static int listener_may_exit = 0;
+static int listener_is_wakeable = 0;/* Pollset supports APR_POLLSET_WAKEABLE */
 static int num_listensocks = 0;
 static apr_int32_t conns_this_child;/* MaxConnectionsPerChild, only access
in listener thread */
@@ -199,6 +202,17 @@ module AP_MODULE_DECLARE_DATA mpm_event_module;
 struct event_srv_cfg_s;
 typedef struct event_srv_cfg_s event_srv_cfg;
 
+/*
+ * The pollset for sockets that are in any of the timeout queues. Currently
+ * we use the timeout_mutex to make sure that connections are added/removed
+ * atomically to/from both event_pollset and a timeout queue. Otherwise
+ * some confusion can happen under high load if timeout queues and pollset
+ * get out of sync.
+ * XXX: It should be possible to make the lock unnecessary in many or even all
+ * XXX: cases.
+ */
+static apr_pollset_t *event_pollset;
+
 struct event_conn_state_t {
 /** APR_RING of expiration timeouts */
 APR_RING_ENTRY(event_conn_state_t) timeout_list;
@@ -244,24 +258,52 @@ static struct timeout_queue *write_completion_q,
 *keepalive_q,
 *linger_q,
 *short_linger_q;
+static apr_time_t queues_next_expiry;
 
 static apr_pollfd_t *listener_pollfd;
 
+/* Prevent extra poll/wakeup calls for timeouts close in the future (queues
+ * have the granularity of a second anyway).
+ * XXX: Wouldn't 0.5s (instead of 0.1s) be "enough"?
+ */
+#define TIMEOUT_FUDGE_FACTOR APR_TIME_C(10) /* 100 ms */
+
+/* Same goal as for TIMEOUT_FUDGE_FACTOR (avoid extra poll calls), but applied
+ * to timers. Since their timeouts are custom (user defined), we can't be too
+ * approximative here (hence using 0.01s).
+ */
+#define EVENT_FUDGE_FACTOR APR_TIME_C(1) /* 10 ms */
+
 /*
  * Macros for accessing struct timeout_queue.
  * For TO_QUEUE_APPEND and TO_QUEUE_REMOVE, timeout_mutex must be held.
  */
-#define TO_QUEUE_APPEND(q, el)\
-do {  \
-APR_RING_INSERT_TAIL(&(q)->head, el, event_conn_state_t,  \
- timeout_list);   \
-++*(q)->total;\
-++(q)->count; \
-} while (0)
+static void TO_QUEUE_APPEND(struct timeout_queue *q, event_conn_state_t *el)
+{
+apr_time_t q_expiry;
 
+APR_RING_INSERT_TAIL(>head, el, event_conn_state_t, timeout_list);
+++*q->total;
+++q->count;
+
+/* Cheaply update the overall queues' next expiry according to the
+ * first entry of this queue (oldest), if necessary.
+ */
+el = APR_RING_FIRST(>head);
+q_expiry = el->queue_timestamp + q->timeout;
+if (!queues_next_expiry
+|| queues_next_expiry > q_expiry + TIMEOUT_FUDGE_FACTOR) {
+VOLATILE_READ(apr_time_t, queues_next_expiry) = q_expiry;
+/* Unblock the poll()ing listener for it to update its timeout. */
+if (listener_is_wakeable) {
+apr_pollset_wakeup(event_pollset);
+}
+}
+}
+
 #define TO_QUEUE_REMOVE(q, el)\
 do {  \
-APR_RING_REMOVE(el, timeout_list);\
+APR_RING_REMOVE((el), timeout_list);  \
 --*(q)->total;\
 --(q)->count; \
 } while (0)
@@ -277,19 +319,9 @@ static apr_pollfd_t *listener_pollfd;
 (q)->next = NULL; \
 } while (0)
 
-#define TO_QUEUE_ELEM_INIT(el) APR_RING_ELEM_INIT(el, timeout_list)
+#define TO_QUEUE_ELEM_INIT(el) \
+APR_RING_ELEM_INIT(el, timeout_list)
 
-/*
- * The pollset 

Re: [proposed] 2.4 Maintenance SIG

[William A Rowe Jr]

On Tue, Jan 3, 2017 at 2:18 AM, Graham Leggett  wrote:
> On 03 Jan 2017, at 2:11 AM, William A Rowe Jr  wrote:
>
>> So I'd like to know, in light of a perpetual chain of (often build and/or 
>> run-time breaking regression) enhancements, if there is support for a 
>> 2.4.24.x release chain during the 3.0 transition? And support for 
>> potentially 3x backports to 2.4.x, 2.4.24.x and 2.2.x, of really serious bug 
>> fixes?
>>
>> It's clear this project doesn't agree, so the question twists to how we 
>> agree to disagree.
>
> Can you clarify the problem you’re trying to solve?

There are multiple recent breakages in httpd. The backport proposal to 2.4
extending mod_dav would have introduced yet another. Other features on
the table such as proxy protocol enhancement, or mod_status extensions
all introduce more and more opportunities to introduce yet another release
with regressions.

I'm wondering if there is anyone interested in a regression-fix-only 2.4.26 that
finally proves to be a workable upgrade for all httpd users? Only bug fixes
which correct defects introduced during 2.4.x would be addressed. Then
skip on to 2.4.27 with new regressions, and repeat the mop-up process in
2.4.28, etc.

Most projects would call these 2.5.0 new features, 2.5.x regression and
bug fixes, 2.6.0 new features, 2.6.x regression and bug fixes etc.

But httpd seems to want to keep a unique numbering schema, because
other numbering conventions weren't invented here.

Is it really unreasonable to expect subsequent releases to not introduce
new errors and become more error-free between enchancement releases?

Re: mod_http2 and Frequent wake-ups for mpm_event

[Stefan Priebe - Profihost AG]

Am 18.01.2017 um 22:53 schrieb Yann Ylavic:
> On Wed, Jan 18, 2017 at 10:50 PM, Yann Ylavic  wrote:
>> On Wed, Jan 18, 2017 at 10:44 PM, Yann Ylavic  wrote:
>>> On Wed, Jan 18, 2017 at 10:23 PM, Stefan Priebe - Profihost AG
>>>  wrote:

>>>
>>> Also, do you confirm that:
>>> 1. it didn't crash with v5 + original (2.4.25's) mod_http2?
>>
>> Hmm, v5 was against 2.4.23, but still no crash happened with it right?
>>
>>> 2. it started crashing with v6 + original mod_http2 (before update to 
>>> v1.8.8)?
> 
> And of course, no crash in 2.4.25 with no patch at all?

Yes as well.

Stefan

Re: mod_http2 and Frequent wake-ups for mpm_event

[Stefan Priebe - Profihost AG]

Am 18.01.2017 um 22:50 schrieb Yann Ylavic:
> On Wed, Jan 18, 2017 at 10:44 PM, Yann Ylavic  wrote:
>> On Wed, Jan 18, 2017 at 10:23 PM, Stefan Priebe - Profihost AG
>>  wrote:
>>>
>>
>> Also, do you confirm that:
>> 1. it didn't crash with v5 + original (2.4.25's) mod_http2?
> 
> Hmm, v5 was against 2.4.23, but still no crash happened with it right?

yes

> 
>> 2. it started crashing with v6 + original mod_http2 (before update to 
>> v1.8.8)?

Re: mod_http2 and Frequent wake-ups for mpm_event

[Yann Ylavic]

On Wed, Jan 18, 2017 at 10:50 PM, Yann Ylavic  wrote:
> On Wed, Jan 18, 2017 at 10:44 PM, Yann Ylavic  wrote:
>> On Wed, Jan 18, 2017 at 10:23 PM, Stefan Priebe - Profihost AG
>>  wrote:
>>>
>>
>> Also, do you confirm that:
>> 1. it didn't crash with v5 + original (2.4.25's) mod_http2?
>
> Hmm, v5 was against 2.4.23, but still no crash happened with it right?
>
>> 2. it started crashing with v6 + original mod_http2 (before update to 
>> v1.8.8)?

And of course, no crash in 2.4.25 with no patch at all?

Re: mod_http2 and Frequent wake-ups for mpm_event

[Yann Ylavic]

On Wed, Jan 18, 2017 at 10:44 PM, Yann Ylavic  wrote:
> On Wed, Jan 18, 2017 at 10:23 PM, Stefan Priebe - Profihost AG
>  wrote:
>>
>
> Also, do you confirm that:
> 1. it didn't crash with v5 + original (2.4.25's) mod_http2?

Hmm, v5 was against 2.4.23, but still no crash happened with it right?

> 2. it started crashing with v6 + original mod_http2 (before update to v1.8.8)?

Re: mod_http2 and Frequent wake-ups for mpm_event

[Stefan Priebe - Profihost AG]

Am 18.01.2017 um 22:44 schrieb Yann Ylavic:
> On Wed, Jan 18, 2017 at 10:23 PM, Stefan Priebe - Profihost AG
>  wrote:
>>
>> sadly it does not solve the issue.
> 
> OK, thanks, back to the paper.
> 
> Any [error] log maybe?
What kind of? server-error.log just says
[Wed Jan 18 22:45:34.050450 2017] [core:notice] [pid 16119:tid
140600654485376] AH00051: child pid 16372 exit signal Segmentation fault
(11), possible coredump in /tmp/

> Also, do you confirm that:
> 1. it didn't crash with v5 + original (2.4.25's) mod_http2?
v5 does not apply to 2.4.25. If you can send me a v5 version that
applies to 2.4.25 i'll test.

> 2. it started crashing with v6 + original mod_http2 (before update to v1.8.8)?
Yes.

Stefan

Re: mod_http2 and Frequent wake-ups for mpm_event

[Yann Ylavic]

On Wed, Jan 18, 2017 at 10:42 PM, Stefan Priebe - Profihost AG
 wrote:
> I also tried to remove rev 1762701 from the v6 patchset. But it does not
> match.

That's more or less what v7 does..

Re: mod_http2 and Frequent wake-ups for mpm_event

[Yann Ylavic]

On Wed, Jan 18, 2017 at 10:23 PM, Stefan Priebe - Profihost AG
 wrote:
>
> sadly it does not solve the issue.

OK, thanks, back to the paper.

Any [error] log maybe?

Also, do you confirm that:
1. it didn't crash with v5 + original (2.4.25's) mod_http2?
2. it started crashing with v6 + original mod_http2 (before update to v1.8.8)?

Re: mod_http2 and Frequent wake-ups for mpm_event

[Stefan Priebe - Profihost AG]

I also tried to remove rev 1762701 from the v6 patchset. But it does not
match.

Stefan
Am 18.01.2017 um 22:23 schrieb Stefan Priebe - Profihost AG:
> Hi Yann,
> 
> sadly it does not solve the issue.
> 
> Trace looks still the same:
> Core was generated by `/usr/local/apache2/bin/httpd -k start'.
> Program terminated with signal SIGSEGV, Segmentation fault.
> #0  0x7fe0228a9014 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
> (gdb) bt
> #0  0x7fe0228a9014 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
> #1  0x7fe022d3c036 in ?? () from /usr/lib/x86_64-linux-gnu/libapr-1.so.0
> #2  0x7fe022d3c46f in apr_hash_set () from
> /usr/lib/x86_64-linux-gnu/libapr-1.so.0
> #3  0x0052a238 in h2_ihash_remove ()
> #4  0x0045 in ?? ()
> #5  0x7fe0022c5328 in ?? ()
> #6  0x7fe002fec300 in ?? ()
> #7  0x00506b24 in purge_stream ()
> #8  0x7fe0014690a0 in ?? ()
> #9  0x7fe000fad2d8 in ?? ()
> #10 0x7fe00194e0a0 in ?? ()
> #11 0x0045 in ?? ()
> #12 0x7fe0014690a0 in ?? ()
> #13 0x7fe000fad2d8 in ?? ()
> #14 0x7fe002fec340 in ?? ()
> #15 0x0052a18f in ihash_iter ()
> #16 0x7fe0014690a0 in ?? ()
> #17 0x0004 in ?? ()
> #18 0x7fe0014690a0 in ?? ()
> #19 0x7fe002fec3c0 in ?? ()
> #20 0x in ?? ()
> 
> Stefan
> 
> Am 18.01.2017 um 17:49 schrieb Yann Ylavic:
>> On Wed, Jan 18, 2017 at 4:38 PM, Stefan Priebe - Profihost AG
>>  wrote:
>>> Ok will wait whether you provide a fix or I should revert that part.
>>
>> Patch updated in PR 57399.
>>
>> Thanks,
>> Yann.
>>

Re: mod_http2 and Frequent wake-ups for mpm_event

[Stefan Priebe - Profihost AG]

Hi Yann,

sadly it does not solve the issue.

Trace looks still the same:
Core was generated by `/usr/local/apache2/bin/httpd -k start'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x7fe0228a9014 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
(gdb) bt
#0  0x7fe0228a9014 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#1  0x7fe022d3c036 in ?? () from /usr/lib/x86_64-linux-gnu/libapr-1.so.0
#2  0x7fe022d3c46f in apr_hash_set () from
/usr/lib/x86_64-linux-gnu/libapr-1.so.0
#3  0x0052a238 in h2_ihash_remove ()
#4  0x0045 in ?? ()
#5  0x7fe0022c5328 in ?? ()
#6  0x7fe002fec300 in ?? ()
#7  0x00506b24 in purge_stream ()
#8  0x7fe0014690a0 in ?? ()
#9  0x7fe000fad2d8 in ?? ()
#10 0x7fe00194e0a0 in ?? ()
#11 0x0045 in ?? ()
#12 0x7fe0014690a0 in ?? ()
#13 0x7fe000fad2d8 in ?? ()
#14 0x7fe002fec340 in ?? ()
#15 0x0052a18f in ihash_iter ()
#16 0x7fe0014690a0 in ?? ()
#17 0x0004 in ?? ()
#18 0x7fe0014690a0 in ?? ()
#19 0x7fe002fec3c0 in ?? ()
#20 0x in ?? ()

Stefan

Am 18.01.2017 um 17:49 schrieb Yann Ylavic:
> On Wed, Jan 18, 2017 at 4:38 PM, Stefan Priebe - Profihost AG
>  wrote:
>> Ok will wait whether you provide a fix or I should revert that part.
> 
> Patch updated in PR 57399.
> 
> Thanks,
> Yann.
> 

Re: Summary: Broken FastCGI with httpd

[Jim Jagielski]

The thing is that when we are proxying back, some of those
values can't make sense, since, for example, there is no
"real" path on the httpd server that maps to the actual
request file

I think the original idea was to simply send the full URL to
the FCGI server, to let it parse things out for itself...
After all, it's easier for the FCGI server to know the SCRIPT_NAME
than httpd to "guess"... This was, iirc, the initial
concept related to how to leverage SCRIPT_FILENAME on
the PHP side.

What we need is some specific understanding between, in this
case, PHP-FPM and httpd on what it wants/expects/needs regarding
these env-vars. Yeah, SCRIPT_FILENAME seems core to this, I
think.

> On Jan 18, 2017, at 2:01 PM, Jacob Champion  wrote:
> 
> On 01/18/2017 06:43 AM, Jim Jagielski wrote:
>> Also, the fact that different methods of invoking FCGI result
>> in different vars, at 1st blush, doesn't seem "incorrect"
>> assuming that each difference makes some sense, in a way.
> 
> They don't make sense. For example, mod_proxy_fcgi can be set up in a way 
> that mirrors the mod_fastcgi preferred configuration, using an Action:
> 
>  AddType application/x-php7-fpm .php
>  Action application/x-php7-fpm /php7-fpm virtual
>  
>SetHandler proxy:fcgi://localhost:9000
>  
> 
> This results in the following variables when I access the URL 
> '/hello.php/path/info?query':
> 
>  SCRIPT_FILENAME: /var/www/php7-fpm
>  SCRIPT_NAME: /php7-fpm
>  PATH_INFO:   /hello.php/path/info
>  PATH_TRANSLATED: /var/www/hello.php/path/info
>  QUERY_STRING:query
> 
> We only get QUERY_STRING right. (Well, I won't say SCRIPT_FILENAME is 
> "wrong", since there is no standard definition, but it's not helpful.) 
> SCRIPT_NAME is supposed to point to a possible script-path (see RFC 3875 for 
> definitions), '/hello.php' in this instance. PATH_INFO and PATH_TRANSLATED 
> should not include 'hello.php' in their values.

That is because we have no idea what SCRIPT_NAME is... With the "guess" that
it is /php7-fpm, then PATH_INFO kind of makes sense, since it is
the portion of the URI following the script. And considering
that we use

   Action application/x-php7-fpm /php7-fpm virtual

the 2nd parameter is specifically noted as *being the cgi-script* and
so of course httpd assumes that /php7-fpm is SCRIPT_NAME, because
we explicitly called it that :)


> 
> It's not just Action. PR 51517 contains examples using ProxyPassMatch.
> 
> --Jacob

Re: Summary: Broken FastCGI with httpd

[Jacob Champion]

On 01/18/2017 06:47 AM, David Zuelke wrote:

Thanks for picking this up!


And thank you for driving this on the PHP side!

--Jacob

Re: Summary: Broken FastCGI with httpd

[Jacob Champion]

On 01/17/2017 11:46 PM, Stefan Eissing wrote:

Would it make sense to narrow down the setups to a few blessed ones that become 
part of our tests?


I think so. Problem is that I don't, myself, know what is actually in 
use out there. I keep learning about new ways to invoke CGI every month, 
it seems.


If anyone has a test matrix they're using for themselves, or even just a 
good list of "ways to configure (Fast)CGI", that'd be really useful.


--Jacob

Re: Summary: Broken FastCGI with httpd

[Jacob Champion]

On 01/18/2017 06:56 AM, Eric Covener wrote:

Unfortunately we might need our own directives for the overrides here
to get them to run after the normal vars are calculated and only when
we're in the proxy_fcgi  handler.


See https://bz.apache.org/bugzilla/show_bug.cgi?id=28903 for an older 
conversation about this.


--Jacob

Re: Summary: Broken FastCGI with httpd

[Jacob Champion]

On 01/18/2017 06:43 AM, Jim Jagielski wrote:

Also, the fact that different methods of invoking FCGI result
in different vars, at 1st blush, doesn't seem "incorrect"
assuming that each difference makes some sense, in a way.


They don't make sense. For example, mod_proxy_fcgi can be set up in a 
way that mirrors the mod_fastcgi preferred configuration, using an Action:


  AddType application/x-php7-fpm .php
  Action application/x-php7-fpm /php7-fpm virtual
  
SetHandler proxy:fcgi://localhost:9000
  

This results in the following variables when I access the URL 
'/hello.php/path/info?query':


  SCRIPT_FILENAME: /var/www/php7-fpm
  SCRIPT_NAME: /php7-fpm
  PATH_INFO:   /hello.php/path/info
  PATH_TRANSLATED: /var/www/hello.php/path/info
  QUERY_STRING:query

We only get QUERY_STRING right. (Well, I won't say SCRIPT_FILENAME is 
"wrong", since there is no standard definition, but it's not helpful.) 
SCRIPT_NAME is supposed to point to a possible script-path (see RFC 3875 
for definitions), '/hello.php' in this instance. PATH_INFO and 
PATH_TRANSLATED should not include 'hello.php' in their values.


It's not just Action. PR 51517 contains examples using ProxyPassMatch.

--Jacob

Re: Async write completion broken in trunk?

[Yann Ylavic]

On Wed, Jan 18, 2017 at 2:43 PM, Graham Leggett  wrote:
> On 18 Jan 2017, at 3:38 PM, Jim Jagielski  wrote:
>
>> That's a good overview... Yeah, it might be getter better, but it does
>> seem that the nature of the bugs implies it's also getting more
>> fragile. Could be, likely is, just my impression due to some weird
>> things I've seen lately on some testing.

We could make no improvement in trunk (bugfixes only) and have better
feeling, yet that's not how we'll make httpd-next...

Let's fix the bugs or revert if things really prove catastrophic, right?

>
> Pipelining support as it stands is really ugly.
>
> There is a point that is reached where we need to ask the question
> “is there more data available to read”, and if the answer is yes, we
> need to follow the pipeline path. We answer the question “is there
> more data available to read” by trying to read some data, and then in
> turn by tying ourselves in ugly knots trying to “unread” that data so
> the next request can work.

Are we talking about ap_check_pipeline()?

>
> I want to fix this properly so we don’t read data, we just get an
> answer “there is data to be read”.

Not sure I get the nuance, but "there is *HTTP* data to be read" (like
ap_check_pipeline() does) looks reasonable to me, unless we want
something like a "TLS close notify" (or trailing CRLFs) start a new
request processing which will fail (or block).


Regards,
Yann.

Re: JSON for mod_status

[William A Rowe Jr]

Really, this is now in the PMC's court. Doug and Aaron designed the BMX
bean structure and module implementation. I'm aware that jfc's crew has
also been a consumer of the module, so it already falls into that multi-vendor,
multi-use case scenario.

I'll leave this to them to advocate for httpd adoption of the work, I was just
the messenger and sometimes-maintainer.

As far as well-defined, the existing 'additonal HTML' structure for mod_status
essentially sucks; there is no way for mod_status to comprehend the data
coming back to it. It simply consists of an HTML text stream. It knows not
one thing about the extended html presented by an arbitrary third party
module.

Moving forwards with your suggestion, it is simple to pass a new 'format'
enum to the modules with the callback, so they know it should be plain
html, modern html, xml, json, or any other representation we want to add.
Modules who don't implement a given value will simply not append their
results in a particular format. So if we want to add a particularly well-
formed xml for xslt representation of status data sometime in the future,
mod_status will not know how to interpret this for other third party mods,
and those that aren't patched for the new 'format' value will simply not
add results.

The bean concept defined to mod_bmx what the data format was. I'm
not sure it was comprehensive enough in terms of arrays of arrays or
other prospective use cases. E.g. could a mod_status_bmx actually
interrogate the bmx providers and offer some legible html status
output today or xml for xslt presentation some day in the future? That's
a good question and I'd like to hear Aaron's or Doug's thoughts.

Bill




On Tue, Jan 17, 2017 at 12:33 PM, Jim Jagielski  wrote:
> It all depends on what Bill decides regarding mod_bmx and if
> it is something we intent to backport to 2.4.x
>
> Still not sure on how to *use* BMX, or how other modules
> "hook in" (right now we have several modules hook into
> mod_status so the "how" is well known and documented), so
> I would require some sort of docs in addition to the actual
> code, of course.
>
>> On Jan 17, 2017, at 12:35 PM, Luca Toscano  wrote:
>>
>>
>>
>> 2016-11-30 18:54 GMT+01:00 Jim Jagielski :
>> I'm thinking about adding JSON support to mod_status...
>> the "plain" version output really stinks and lacks parity
>> w/ the info we provide via HTML, and it would be nice
>> to produce a really easily parseable format.
>>
>> Thoughts...?
>>
>> I know it was extensively discussed, but do we have an agreement about a 
>> plan to add this feature?  :)
>>
>> Luca
>>
>>
>>
>

Re: mod_http2 and Frequent wake-ups for mpm_event

[Yann Ylavic]

On Wed, Jan 18, 2017 at 4:38 PM, Stefan Priebe - Profihost AG
 wrote:
> Ok will wait whether you provide a fix or I should revert that part.

Patch updated in PR 57399.

Thanks,
Yann.

ApacheCon CFP closing soon (11 February)

[Rich Bowen]

Hello, fellow Apache enthusiast. Thanks for your participation, and
interest in, the projects of the Apache Software Foundation.

I wanted to remind you that the Call For Papers (CFP) for ApacheCon
North America, and Apache: Big Data North America, closes in less than a
month. If you've been putting it off because there was lots of time
left, it's time to dig for that inspiration and get those talk proposals in.

It's also time to discuss with your developer and user community whether
there's a track of talks that you might want to propose, so that you
have more complete coverage of your project than a talk or two.

We're looking for talks directly, and indirectly, related to projects at
the Apache Software Foundation. These can be anything from in-depth
technical discussions of the projects you work with, to talks about
community, documentation, legal issues, marketing, and so on. We're also
very interested in talks about projects and services built on top of
Apache projects, and case studies of how you use Apache projects to
solve real-world problems.

We are particularly interested in presentations from Apache projects
either in the Incubator, or recently graduated. ApacheCon is where
people come to find out what technology they'll be using this time next
year.

Important URLs are:

To submit a talk for Apache: Big Data -
http://events.linuxfoundation.org/events/apache-big-data-north-america/program/cfp
To submit a talk for ApacheCon -
http://events.linuxfoundation.org/events/apachecon-north-america/program/cfp

To register for Apache: Big Data -
http://events.linuxfoundation.org/events/apache-big-data-north-america/attend/register-
To register for ApacheCon -
http://events.linuxfoundation.org/events/apachecon-north-america/attend/register-

Early Bird registration rates end March 12th, but if you're a committer
on an Apache project, you get the low committer rate, which is less than
half of the early bird rate!

For further updated about ApacheCon, follow us on Twitter, @ApacheCon,
or drop by our IRC channel, #apachecon on the Freenode IRC network. Or
contact me - rbo...@apache.org - with any questions or concerns.

Thanks!

Rich Bowen, VP Conferences, Apache Software Foundation

-- 
(You've received this email because you're on a dev@ or users@ mailing
list of an Apache Software Foundation project. For subscription and
unsubscription information, consult the headers of this email message,
as this varies from one list to another.)

Re: mod_http2 and Frequent wake-ups for mpm_event

[Stefan Priebe - Profihost AG]

Ok will wait whether you provide a fix or I should revert that part.

Stefan

Excuse my typo sent from my mobile phone.

> Am 18.01.2017 um 16:32 schrieb Yann Ylavic :
> 
> On Wed, Jan 18, 2017 at 4:17 PM, Stefan Priebe - Profihost AG
>  wrote:
>> 
>> is it enought to revert that commit?
> 
> I think so (worth a try at least).
> I'm about to commit sort of this revert to trunk, with a possible 
> explanation...
> 
> Thanks Stefan for all these follow ups.

Re: mod_http2 and Frequent wake-ups for mpm_event

[Yann Ylavic]

On Wed, Jan 18, 2017 at 4:17 PM, Stefan Priebe - Profihost AG
 wrote:
>
> is it enought to revert that commit?

I think so (worth a try at least).
I'm about to commit sort of this revert to trunk, with a possible explanation...

Thanks Stefan for all these follow ups.

Re: mod_http2 and Frequent wake-ups for mpm_event

[Stefan Priebe - Profihost AG]

Hi Yann,

is it enought to revert that commit?
http://svn.apache.org/r1762701

Stefan

Am 18.01.2017 um 12:49 schrieb Stefan Priebe - Profihost AG:
> Hi,
> 
> site is mostly used with http2. So it may be totally unrelated to
> mod_http2. Sorry for the noise than. I just thought it by digging
> through the traces.
> 
> Stefan
> 
> Am 18.01.2017 um 12:34 schrieb Yann Ylavic:
>> Hi Stefan,
>>
>> On Wed, Jan 18, 2017 at 11:33 AM, Stefan Priebe - Profihost AG
>>  wrote:
>>>
>>> after applying the event patch to 2.4.25 from
>>> https://bz.apache.org/bugzilla/show_bug.cgi?id=57399.
>>>
>>> I'm seeing segfaults in the mod_http2 code. I already bumped mod_http2
>>> to v1.8.8. But the segfaults are still happening.
>>
>> Does it happen only with http2, or maybe your site is mostly h2-used only?
>>

Re: Summary: Broken FastCGI with httpd

[Jim Jagielski]

Just some additional info (the perl script described might be useful,
esp if we fold it into the test framework):

https://bugs.php.net/bug.php?id=54152


> On Jan 18, 2017, at 9:47 AM, David Zuelke  wrote:
> 
>> On 17.01.2017, at 23:16, Jacob Champion  wrote:
>> 
>> (This conversation is currently spread over Bugzilla, IRC, GitHub, and 
>> php-internals. Here's my attempt at summarizing it for all of you. If you 
>> have no interest in CGI or FastCGI, stop reading now.)
> 
> Thanks for picking this up!
> 
> 
>> 2) Define what SCRIPT_FILENAME means.
>> 
>> SCRIPT_FILENAME isn't actually a CGI 1.1 standard variable. We appear to 
>> have defined it as "whatever r->filename contains", so we've effectively 
>> coupled our implementation details to external clients. PHP-FPM and 
>> fcgiwrap, for example, assume that SCRIPT_FILENAME should point to the 
>> script that should be executed to handle the request. We need to standardize 
>> it.
> 
> There's one more caveat around SCRIPT_FILENAME, I think: it might not be the 
> same for httpd and the FCGI backend if they're running on separate machines!
> 
> David
> 

Re: Summary: Broken FastCGI with httpd

[Eric Covener]

On Wed, Jan 18, 2017 at 9:47 AM, David Zuelke  wrote:
> There's one more caveat around SCRIPT_FILENAME, I think: it might not be the 
> same for httpd and the FCGI backend if they're running on separate machines!

I think this goes to the notion of overriding these variables w/
config rather than trying to coax the server to try to construct them
correctly by rewriting.

Unfortunately we might need our own directives for the overrides here
to get them to run after the normal vars are calculated and only when
we're in the proxy_fcgi  handler.

-- 
Eric Covener
cove...@gmail.com

Re: JSON for mod_status

[Daniel Gruno]

On 01/18/2017 03:19 PM, Luca Toscano wrote:
> 
> 
> 2017-01-18 10:56 GMT+01:00 Daniel Gruno  >:
> 
> On 01/17/2017 07:33 PM, Jim Jagielski wrote:
> > It all depends on what Bill decides regarding mod_bmx and if
> > it is something we intent to backport to 2.4.x
> >
> > Still not sure on how to *use* BMX, or how other modules
> > "hook in" (right now we have several modules hook into
> > mod_status so the "how" is well known and documented), so
> > I would require some sort of docs in addition to the actual
> > code, of course.
> 
> Some JFDI in the meantime; https://httpd.apache.org/server-status
>  :)
> JSON: http://httpd.apache.org/server-status?view=json_status
> 
> 
> 
> Really nice work! I'd really like to see (if the author agrees :P) some
> mod_lua scripts shipped with the httpd release and referenced in our
> documentation. 

Hey, it's ALv2, always has been - if we wanna ship it, so be it :)
It's at https://github.com/Humbedooh/server-status at the moment, and
needs a tiny bit of cleanup if it's to be shipped.

With regards,
Daniel.

> 
> Luca
> 

Re: Summary: Broken FastCGI with httpd

[David Zuelke]

> On 17.01.2017, at 23:16, Jacob Champion  wrote:
> 
> (This conversation is currently spread over Bugzilla, IRC, GitHub, and 
> php-internals. Here's my attempt at summarizing it for all of you. If you 
> have no interest in CGI or FastCGI, stop reading now.)

Thanks for picking this up!


> 2) Define what SCRIPT_FILENAME means.
> 
> SCRIPT_FILENAME isn't actually a CGI 1.1 standard variable. We appear to have 
> defined it as "whatever r->filename contains", so we've effectively coupled 
> our implementation details to external clients. PHP-FPM and fcgiwrap, for 
> example, assume that SCRIPT_FILENAME should point to the script that should 
> be executed to handle the request. We need to standardize it.

There's one more caveat around SCRIPT_FILENAME, I think: it might not be the 
same for httpd and the FCGI backend if they're running on separate machines!

David

Re: Summary: Broken FastCGI with httpd

[Jim Jagielski]

It seems to me that SCRIPT_FILENAME is the key w/ PHP-FPM, but
I could be wrong.

Also, the fact that different methods of invoking FCGI result
in different vars, at 1st blush, doesn't seem "incorrect"
assuming that each difference makes some sense, in a way.

Finally, I think that instead of trying to address these in
the various proxy modules/submodules, etc, we should do so
in util_script.c instead. One central place where we "canonically"
set these params, and handle the different ways r->filename may
be mangled ;)

Re: JSON for mod_status

[Luca Toscano]

2017-01-18 10:56 GMT+01:00 Daniel Gruno :

> On 01/17/2017 07:33 PM, Jim Jagielski wrote:
> > It all depends on what Bill decides regarding mod_bmx and if
> > it is something we intent to backport to 2.4.x
> >
> > Still not sure on how to *use* BMX, or how other modules
> > "hook in" (right now we have several modules hook into
> > mod_status so the "how" is well known and documented), so
> > I would require some sort of docs in addition to the actual
> > code, of course.
>
> Some JFDI in the meantime; https://httpd.apache.org/server-status :)
> JSON: http://httpd.apache.org/server-status?view=json_status
>
>
Really nice work! I'd really like to see (if the author agrees :P) some
mod_lua scripts shipped with the httpd release and referenced in our
documentation.

Luca

Re: Async write completion broken in trunk?

[Graham Leggett]

On 18 Jan 2017, at 3:38 PM, Jim Jagielski  wrote:

> That's a good overview... Yeah, it might be getter better, but it does
> seem that the nature of the bugs implies it's also getting more
> fragile. Could be, likely is, just my impression due to some weird
> things I've seen lately on some testing.

Pipelining support as it stands is really ugly.

There is a point that is reached where we need to ask the question “is there 
more data available to read”, and if the answer is yes, we need to follow the 
pipeline path. We answer the question “is there more data available to read” by 
trying to read some data, and then in turn by tying ourselves in ugly knots 
trying to “unread” that data so the next request can work.

I want to fix this properly so we don’t read data, we just get an answer “there 
is data to be read”. This will take a bit of thought, but doesn’t seem 
difficult to do.

Regards,
Graham
—



smime.p7s
Description: S/MIME cryptographic signature

Re: Async write completion broken in trunk?

[Jim Jagielski]

> On Jan 18, 2017, at 8:25 AM, Luca Toscano  wrote:
> 
> 
> 
> 2017-01-18 14:00 GMT+01:00 Jim Jagielski :
> 
> > On Jan 18, 2017, at 7:50 AM, Graham Leggett  wrote:
> >
> > On 17 Jan 2017, at 7:40 PM, Luca Toscano  wrote:
> >
> >> Since this email thread seems important, is there any update from anybody 
> >> working on it? It would be great to open a bugzilla task otherwise, to 
> >> track everything and make sure that we don't forget about it :)
> >
> > I looked at this a while back - I found that pipelining support is causing 
> > the blocking.
> >
> > When we’re handling pipelined requests and we reach a limit, we block. What 
> > we should be doing instead is looping round back to WRITE_COMPLETION, 
> > waiting for permission to write again. This should be reasonably 
> > straightforward to fix, but my financial year end is the end of this month 
> > so can’t look at this till February.
> >
> > I suspect pipelining support has been suppressed in v2.4.x event MPM, and 
> > was at some point “fixed” on trunk, bringing this problem back.
> >
> 
> Somewhat on-topic but also off-topic as well, but it does seem
> that event on trunk is getting much more fragile instead of more
> stable. It seems to be picking up some kruft which is making
> event on trunk a *worse* option than what's in 2.4, despite a deeper
> async alignment.
> 
> 
> My 2c view on this is that mpm-event has been getting better during the last 
> months, but the nature of the changes, together with the fact that we have 
> not developed a good way to test regressions, causes long running debugging 
> tasks. In the case of the wake-ups, we are basically testing a new big 
> feature thanks to live traffic from Stefan Priebe (thanks again for the help 
> btw!), meanwhile it would be really great to have some sort of test to fake 
> traffic and spot problems (that would absolutely not substitute the 
> invaluable user feedback of course).
> 

That's a good overview... Yeah, it might be getter better, but it does
seem that the nature of the bugs implies it's also getting more
fragile. Could be, likely is, just my impression due to some weird
things I've seen lately on some testing.

Re: Async write completion broken in trunk?

[Luca Toscano]

2017-01-18 13:50 GMT+01:00 Graham Leggett :

> On 17 Jan 2017, at 7:40 PM, Luca Toscano  wrote:
>
> > Since this email thread seems important, is there any update from
> anybody working on it? It would be great to open a bugzilla task otherwise,
> to track everything and make sure that we don't forget about it :)
>
> I looked at this a while back - I found that pipelining support is causing
> the blocking.
>
> When we’re handling pipelined requests and we reach a limit, we block.
> What we should be doing instead is looping round back to WRITE_COMPLETION,
> waiting for permission to write again. This should be reasonably
> straightforward to fix, but my financial year end is the end of this month
> so can’t look at this till February.
>
> I suspect pipelining support has been suppressed in v2.4.x event MPM, and
> was at some point “fixed” on trunk, bringing this problem back.
>
>
Thanks a lot for the explanation, I didn't mean to rush you but just to
ping the list to figure out if anything new came up :)

Luca

Re: Async write completion broken in trunk?

[Luca Toscano]

2017-01-18 14:00 GMT+01:00 Jim Jagielski :

>
> > On Jan 18, 2017, at 7:50 AM, Graham Leggett  wrote:
> >
> > On 17 Jan 2017, at 7:40 PM, Luca Toscano  wrote:
> >
> >> Since this email thread seems important, is there any update from
> anybody working on it? It would be great to open a bugzilla task otherwise,
> to track everything and make sure that we don't forget about it :)
> >
> > I looked at this a while back - I found that pipelining support is
> causing the blocking.
> >
> > When we’re handling pipelined requests and we reach a limit, we block.
> What we should be doing instead is looping round back to WRITE_COMPLETION,
> waiting for permission to write again. This should be reasonably
> straightforward to fix, but my financial year end is the end of this month
> so can’t look at this till February.
> >
> > I suspect pipelining support has been suppressed in v2.4.x event MPM,
> and was at some point “fixed” on trunk, bringing this problem back.
> >
>
> Somewhat on-topic but also off-topic as well, but it does seem
> that event on trunk is getting much more fragile instead of more
> stable. It seems to be picking up some kruft which is making
> event on trunk a *worse* option than what's in 2.4, despite a deeper
> async alignment.
>
>
My 2c view on this is that mpm-event has been getting better during the
last months, but the nature of the changes, together with the fact that we
have not developed a good way to test regressions, causes long running
debugging tasks. In the case of the wake-ups, we are basically testing a
new big feature thanks to live traffic from Stefan Priebe (thanks again for
the help btw!), meanwhile it would be really great to have some sort of
test to fake traffic and spot problems (that would absolutely not
substitute the invaluable user feedback of course).

Luca

Re: Async write completion broken in trunk?

[Jim Jagielski]

> On Jan 18, 2017, at 8:14 AM, Graham Leggett  wrote:
> 
> On 18 Jan 2017, at 3:05 PM, Jim Jagielski  wrote:
> 
>> Well, there's this, and it seems like there are issues w/
>> the current mutexing as well, causing weird wake-ups.
> 
> Only solution really is to find the bugs and fix them.
> 

Yep... I'm trying to do some stress testing

Re: Async write completion broken in trunk?

[Graham Leggett]

On 18 Jan 2017, at 3:05 PM, Jim Jagielski  wrote:

> Well, there's this, and it seems like there are issues w/
> the current mutexing as well, causing weird wake-ups.

Only solution really is to find the bugs and fix them.

Regards,
Graham
—



smime.p7s
Description: S/MIME cryptographic signature

Re: Async write completion broken in trunk?

[Jim Jagielski]

> On Jan 18, 2017, at 8:01 AM, Graham Leggett  wrote:
> 
> On 18 Jan 2017, at 3:00 PM, Jim Jagielski  wrote:
> 
>> Somewhat on-topic but also off-topic as well, but it does seem
>> that event on trunk is getting much more fragile instead of more
>> stable. It seems to be picking up some kruft which is making
>> event on trunk a *worse* option than what's in 2.4, despite a deeper
>> async alignment.
> 
> Can you give an example?
> 

Well, there's this, and it seems like there are issues w/
the current mutexing as well, causing weird wake-ups.

Re: Async write completion broken in trunk?

[Graham Leggett]

On 18 Jan 2017, at 3:00 PM, Jim Jagielski  wrote:

> Somewhat on-topic but also off-topic as well, but it does seem
> that event on trunk is getting much more fragile instead of more
> stable. It seems to be picking up some kruft which is making
> event on trunk a *worse* option than what's in 2.4, despite a deeper
> async alignment.

Can you give an example?

Regards,
Graham
—



smime.p7s
Description: S/MIME cryptographic signature

Re: Async write completion broken in trunk?

[Jim Jagielski]

> On Jan 18, 2017, at 7:50 AM, Graham Leggett  wrote:
> 
> On 17 Jan 2017, at 7:40 PM, Luca Toscano  wrote:
> 
>> Since this email thread seems important, is there any update from anybody 
>> working on it? It would be great to open a bugzilla task otherwise, to track 
>> everything and make sure that we don't forget about it :)
> 
> I looked at this a while back - I found that pipelining support is causing 
> the blocking.
> 
> When we’re handling pipelined requests and we reach a limit, we block. What 
> we should be doing instead is looping round back to WRITE_COMPLETION, waiting 
> for permission to write again. This should be reasonably straightforward to 
> fix, but my financial year end is the end of this month so can’t look at this 
> till February.
> 
> I suspect pipelining support has been suppressed in v2.4.x event MPM, and was 
> at some point “fixed” on trunk, bringing this problem back.
> 

Somewhat on-topic but also off-topic as well, but it does seem
that event on trunk is getting much more fragile instead of more
stable. It seems to be picking up some kruft which is making
event on trunk a *worse* option than what's in 2.4, despite a deeper
async alignment.

Re: Async write completion broken in trunk?

[Graham Leggett]

On 17 Jan 2017, at 7:40 PM, Luca Toscano  wrote:

> Since this email thread seems important, is there any update from anybody 
> working on it? It would be great to open a bugzilla task otherwise, to track 
> everything and make sure that we don't forget about it :)

I looked at this a while back - I found that pipelining support is causing the 
blocking.

When we’re handling pipelined requests and we reach a limit, we block. What we 
should be doing instead is looping round back to WRITE_COMPLETION, waiting for 
permission to write again. This should be reasonably straightforward to fix, 
but my financial year end is the end of this month so can’t look at this till 
February.

I suspect pipelining support has been suppressed in v2.4.x event MPM, and was 
at some point “fixed” on trunk, bringing this problem back.

Regards,
Graham
—



smime.p7s
Description: S/MIME cryptographic signature

Re: mod_http2 and Frequent wake-ups for mpm_event

[Stefan Eissing]

> Am 18.01.2017 um 13:39 schrieb Stefan Eissing :
> 
> There is currently no cleanup register for shutting down that way. If the 
> main connection pool gets cleaned before the special meta is destroyed, these 
> problem might happen. Not sure.

I am babbling. Of course there is. Question is: does it work correctly when the 
order changes...

Stefan Eissing

bytes GmbH
Hafenstrasse 16
48155 Münster
www.greenbytes.de

Re: mod_http2 and Frequent wake-ups for mpm_event

[Stefan Eissing]

Yann,

all these traces point to the connection shutdown case for mod_http2 which is 
triggered by writing a special meta bucket and cleaning up on its destruction. 
That's the trick to make sure that all data has been sent and similar to the 
EOR bucket. 

There is currently no cleanup register for shutting down that way. If the main 
connection pool gets cleaned before the special meta is destroyed, these 
problem might happen. Not sure.

-Stefan

> Am 18.01.2017 um 13:32 schrieb Stefan Priebe - Profihost AG 
> :
> 
> Hi,
> 
> and a new one just some seconds ago:
> (gdb) bt
> #0  0x7f32a3747014 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
> #1  0x7f32a3bda036 in ?? () from /usr/lib/x86_64-linux-gnu/libapr-1.so.0
> #2  0x7f32a3bda46f in apr_hash_set () from
> /usr/lib/x86_64-linux-gnu/libapr-1.so.0
> #3  0x0052a238 in h2_ihash_remove ()
> #4  0x00b7 in ?? ()
> #5  0x7f329410e328 in ?? ()
> #6  0x7f3283fe6300 in ?? ()
> #7  0x00506b24 in purge_stream ()
> #8  0x7f3282edf0a0 in ?? ()
> #9  0x7f3282ecf2d8 in ?? ()
> #10 0x7f32940e60a0 in ?? ()
> #11 0x00b7 in ?? ()
> #12 0x7f3282edf0a0 in ?? ()
> #13 0x7f3282ecf2d8 in ?? ()
> #14 0x7f3283fe6340 in ?? ()
> #15 0x0052a18f in ihash_iter ()
> #16 0x7f3282edf0a0 in ?? ()
> #17 0x0004 in ?? ()
> #18 0x7f3282edf0a0 in ?? ()
> #19 0x7f3283fe63c0 in ?? ()
> #20 0x in ?? ()
> 
> Stefan
> 
> Am 18.01.2017 um 12:49 schrieb Stefan Priebe - Profihost AG:
>> Hi,
>> 
>> site is mostly used with http2. So it may be totally unrelated to
>> mod_http2. Sorry for the noise than. I just thought it by digging
>> through the traces.
>> 
>> Stefan
>> 
>> Am 18.01.2017 um 12:34 schrieb Yann Ylavic:
>>> Hi Stefan,
>>> 
>>> On Wed, Jan 18, 2017 at 11:33 AM, Stefan Priebe - Profihost AG
>>>  wrote:
 
 after applying the event patch to 2.4.25 from
 https://bz.apache.org/bugzilla/show_bug.cgi?id=57399.
 
 I'm seeing segfaults in the mod_http2 code. I already bumped mod_http2
 to v1.8.8. But the segfaults are still happening.
>>> 
>>> Does it happen only with http2, or maybe your site is mostly h2-used only?
>>> 

Stefan Eissing

bytes GmbH
Hafenstrasse 16
48155 Münster
www.greenbytes.de

Re: mod_http2 and Frequent wake-ups for mpm_event

[Stefan Priebe - Profihost AG]

Hi,

and a new one just some seconds ago:
(gdb) bt
#0  0x7f32a3747014 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#1  0x7f32a3bda036 in ?? () from /usr/lib/x86_64-linux-gnu/libapr-1.so.0
#2  0x7f32a3bda46f in apr_hash_set () from
/usr/lib/x86_64-linux-gnu/libapr-1.so.0
#3  0x0052a238 in h2_ihash_remove ()
#4  0x00b7 in ?? ()
#5  0x7f329410e328 in ?? ()
#6  0x7f3283fe6300 in ?? ()
#7  0x00506b24 in purge_stream ()
#8  0x7f3282edf0a0 in ?? ()
#9  0x7f3282ecf2d8 in ?? ()
#10 0x7f32940e60a0 in ?? ()
#11 0x00b7 in ?? ()
#12 0x7f3282edf0a0 in ?? ()
#13 0x7f3282ecf2d8 in ?? ()
#14 0x7f3283fe6340 in ?? ()
#15 0x0052a18f in ihash_iter ()
#16 0x7f3282edf0a0 in ?? ()
#17 0x0004 in ?? ()
#18 0x7f3282edf0a0 in ?? ()
#19 0x7f3283fe63c0 in ?? ()
#20 0x in ?? ()

Stefan

Am 18.01.2017 um 12:49 schrieb Stefan Priebe - Profihost AG:
> Hi,
> 
> site is mostly used with http2. So it may be totally unrelated to
> mod_http2. Sorry for the noise than. I just thought it by digging
> through the traces.
> 
> Stefan
> 
> Am 18.01.2017 um 12:34 schrieb Yann Ylavic:
>> Hi Stefan,
>>
>> On Wed, Jan 18, 2017 at 11:33 AM, Stefan Priebe - Profihost AG
>>  wrote:
>>>
>>> after applying the event patch to 2.4.25 from
>>> https://bz.apache.org/bugzilla/show_bug.cgi?id=57399.
>>>
>>> I'm seeing segfaults in the mod_http2 code. I already bumped mod_http2
>>> to v1.8.8. But the segfaults are still happening.
>>
>> Does it happen only with http2, or maybe your site is mostly h2-used only?
>>

Re: JSON for mod_status

[Stefan Eissing]

Nice!

> Am 18.01.2017 um 12:46 schrieb Jim Jagielski :
> 
> This is too awesome for words :)
> 
>> On Jan 18, 2017, at 4:56 AM, Daniel Gruno  wrote:
>> 
>> On 01/17/2017 07:33 PM, Jim Jagielski wrote:
>>> It all depends on what Bill decides regarding mod_bmx and if
>>> it is something we intent to backport to 2.4.x
>>> 
>>> Still not sure on how to *use* BMX, or how other modules
>>> "hook in" (right now we have several modules hook into
>>> mod_status so the "how" is well known and documented), so
>>> I would require some sort of docs in addition to the actual
>>> code, of course.
>> 
>> Some JFDI in the meantime; https://httpd.apache.org/server-status :)
>> JSON: http://httpd.apache.org/server-status?view=json_status
>> 
>> With regards,
>> Daniel.
>> 
>>> 
 On Jan 17, 2017, at 12:35 PM, Luca Toscano  wrote:
 
 
 
 2016-11-30 18:54 GMT+01:00 Jim Jagielski :
 I'm thinking about adding JSON support to mod_status...
 the "plain" version output really stinks and lacks parity
 w/ the info we provide via HTML, and it would be nice
 to produce a really easily parseable format.
 
 Thoughts...?
 
 I know it was extensively discussed, but do we have an agreement about a 
 plan to add this feature?  :)
 
 Luca
 
 
 
>>> 
>> 
> 

Stefan Eissing

bytes GmbH
Hafenstrasse 16
48155 Münster
www.greenbytes.de

Re: JSON for mod_status

[Jim Jagielski]

This is too awesome for words :)

> On Jan 18, 2017, at 4:56 AM, Daniel Gruno  wrote:
> 
> On 01/17/2017 07:33 PM, Jim Jagielski wrote:
>> It all depends on what Bill decides regarding mod_bmx and if
>> it is something we intent to backport to 2.4.x
>> 
>> Still not sure on how to *use* BMX, or how other modules
>> "hook in" (right now we have several modules hook into
>> mod_status so the "how" is well known and documented), so
>> I would require some sort of docs in addition to the actual
>> code, of course.
> 
> Some JFDI in the meantime; https://httpd.apache.org/server-status :)
> JSON: http://httpd.apache.org/server-status?view=json_status
> 
> With regards,
> Daniel.
> 
>> 
>>> On Jan 17, 2017, at 12:35 PM, Luca Toscano  wrote:
>>> 
>>> 
>>> 
>>> 2016-11-30 18:54 GMT+01:00 Jim Jagielski :
>>> I'm thinking about adding JSON support to mod_status...
>>> the "plain" version output really stinks and lacks parity
>>> w/ the info we provide via HTML, and it would be nice
>>> to produce a really easily parseable format.
>>> 
>>> Thoughts...?
>>> 
>>> I know it was extensively discussed, but do we have an agreement about a 
>>> plan to add this feature?  :)
>>> 
>>> Luca
>>> 
>>> 
>>> 
>> 
> 

Re: mod_http2 and Frequent wake-ups for mpm_event

[Stefan Priebe - Profihost AG]

Hi,

site is mostly used with http2. So it may be totally unrelated to
mod_http2. Sorry for the noise than. I just thought it by digging
through the traces.

Stefan

Am 18.01.2017 um 12:34 schrieb Yann Ylavic:
> Hi Stefan,
> 
> On Wed, Jan 18, 2017 at 11:33 AM, Stefan Priebe - Profihost AG
>  wrote:
>>
>> after applying the event patch to 2.4.25 from
>> https://bz.apache.org/bugzilla/show_bug.cgi?id=57399.
>>
>> I'm seeing segfaults in the mod_http2 code. I already bumped mod_http2
>> to v1.8.8. But the segfaults are still happening.
> 
> Does it happen only with http2, or maybe your site is mostly h2-used only?
> 

Re: mod_http2 and Frequent wake-ups for mpm_event

[Yann Ylavic]

Hi Stefan,

On Wed, Jan 18, 2017 at 11:33 AM, Stefan Priebe - Profihost AG
 wrote:
>
> after applying the event patch to 2.4.25 from
> https://bz.apache.org/bugzilla/show_bug.cgi?id=57399.
>
> I'm seeing segfaults in the mod_http2 code. I already bumped mod_http2
> to v1.8.8. But the segfaults are still happening.

Does it happen only with http2, or maybe your site is mostly h2-used only?

Re: mod_http2 and Frequent wake-ups for mpm_event

[Yann Ylavic]

On Wed, Jan 18, 2017 at 12:09 PM, Stefan Eissing
 wrote:
> Can't have it all, Stefan!

:)

>
> Seriously, will have a look at this. Thanks for the traces.

I possibly messed up with event's locking in latest patch from PR 57399.
Actually it removes locks around pollset_add/del ([1]) which may be
the cause, though I don't see why for now...

Will take a look too.

[1] http://svn.apache.org/r1762701

Re: mod_http2 and Frequent wake-ups for mpm_event

[Stefan Eissing]

Can't have it all, Stefan!

Seriously, will have a look at this. Thanks for the traces.

-Stefan

> Am 18.01.2017 um 11:33 schrieb Stefan Priebe - Profihost AG 
> :
> 
> Hi Stefan,
> Hi Yann,
> 
> after applying the event patch to 2.4.25 from
> https://bz.apache.org/bugzilla/show_bug.cgi?id=57399.
> 
> I'm seeing segfaults in the mod_http2 code. I already bumped mod_http2
> to v1.8.8. But the segfaults are still happening.
> 
> gdb shows this:
> Core was generated by `/usr/local/apache2/bin/httpd -k start'.
> Program terminated with signal SIGSEGV, Segmentation fault.
> #0  0x7fb75242cd45 in apr_pool_cleanup_kill () from
> /usr/lib/x86_64-linux-gnu/libapr-1.so.0
> (gdb) bt
> #0  0x7fb75242cd45 in apr_pool_cleanup_kill () from
> /usr/lib/x86_64-linux-gnu/libapr-1.so.0
> #1  0x7fb75242ce21 in apr_pool_cleanup_run () from
> /usr/lib/x86_64-linux-gnu/libapr-1.so.0
> #2  0x0051f268 in stream_pool_cleanup ()
> #3  0x7fb7327eb270 in ?? ()
> #4  0x7fb730c180a0 in ?? ()
> #5  0x7fb7327eb294 in ?? ()
> #6  0x7fb7309046c0 in ?? ()
> #7  0x7fb730c180a0 in ?? ()
> #8  0x30c18028 in ?? ()
> #9  0x7fb730c18028 in ?? ()
> #10 0x7fb75242b9be in apr_pool_destroy () from
> /usr/lib/x86_64-linux-gnu/libapr-1.so.0
> #11 0x0051fe61 in h2_stream_destroy ()
> #12 0x002d317482d8 in ?? ()
> #13 0x7fb730c180a0 in ?? ()
> #14 0x7fb7327eb300 in ?? ()
> #15 0x00507ab7 in stream_done ()
> #16 0x in ?? ()
> 
> 
> and or this:
> Program terminated with signal SIGSEGV, Segmentation fault.
> #0  0x7fd2e9034014 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
> (gdb) bt
> #0  0x7fd2e9034014 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
> #1  0x7fd2e94c7036 in ?? () from /usr/lib/x86_64-linux-gnu/libapr-1.so.0
> #2  0x7fd2e94c746f in apr_hash_set () from
> /usr/lib/x86_64-linux-gnu/libapr-1.so.0
> #3  0x0052a238 in h2_ihash_remove ()
> #4  0x004b in ?? ()
> #5  0x7fd2c8d60328 in ?? ()
> #6  0x7fd2c97e9300 in ?? ()
> #7  0x00506b24 in purge_stream ()
> #8  0x7fd2d82140a0 in ?? ()
> #9  0x7fd2c831e2d8 in ?? ()
> #10 0x7fd20352c0a0 in ?? ()
> #11 0x004b in ?? ()
> #12 0x7fd2d82140a0 in ?? ()
> #13 0x7fd2c831e2d8 in ?? ()
> #14 0x7fd2c97e9340 in ?? ()
> #15 0x0052a18f in ihash_iter ()
> #16 0x7fd2d82140a0 in ?? ()
> #17 0x0004 in ?? ()
> #18 0x7fd2d82140a0 in ?? ()
> #19 0x7fd2c97e93c0 in ?? ()
> #20 0x in ?? ()
> 
> 
> Greets,
> Stefan
> 
> Am 17.01.2017 um 21:53 schrieb Stefan Priebe:
>> Hi Yann,
>> 
>> while testing V6 i'm experiencing segfaults.
>> 
>> exit signal Segmentation
>> 
>> server-error.log:
>> AH00052: child pid 14110 exit signal Segmentation fault (11)
>> 
>> currently i'm trying to grab a core dump.
>> 
>> Greets,
>> Stefan
>> 
>> Am 26.12.2016 um 21:18 schrieb Stefan Priebe - Profihost AG:
>>> Am 23.12.2016 um 01:41 schrieb Yann Ylavic:
 Hi Stefan,
 
 On Tue, Dec 20, 2016 at 1:52 PM, Stefan Priebe - Profihost AG
  wrote:
> 
> Today i had another server giving no answers to any requests. apache
> fullstatus did not respond.
 
 Since v5 of the patch, I committed another related change in trunk,
 namely: http://svn.apache.org/r1774538
 It's about lingering keepalive connections on graceful restart which
 may not cause a wakeup.
 Does it help?
>>> 
>>> I'll try that but wanted to rebuild based on http 2.4.25. But your
>>> mpm_event_listener_wakeup_bug57399_V5 patch does no longer apply to http
>>> 2.2.25. Can you rebase it?
>>> 
> gdb bt shows this for all httpd childs:
 
 These backtraces are the ones of the main thread, probably not the
 culprit.
 What does "thread apply all bt" say?
>>> 
>>> Will redo / save that output next time.
>>> 
>>> Thanks!
>>> 
>>> Greets,
>>> Stefan
>>> 
 Regards,
 Yann.
 

Stefan Eissing

bytes GmbH
Hafenstrasse 16
48155 Münster
www.greenbytes.de

mod_http2 and Frequent wake-ups for mpm_event

[Stefan Priebe - Profihost AG]

Hi Stefan,
 Hi Yann,

after applying the event patch to 2.4.25 from
https://bz.apache.org/bugzilla/show_bug.cgi?id=57399.

I'm seeing segfaults in the mod_http2 code. I already bumped mod_http2
to v1.8.8. But the segfaults are still happening.

gdb shows this:
Core was generated by `/usr/local/apache2/bin/httpd -k start'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x7fb75242cd45 in apr_pool_cleanup_kill () from
/usr/lib/x86_64-linux-gnu/libapr-1.so.0
(gdb) bt
#0  0x7fb75242cd45 in apr_pool_cleanup_kill () from
/usr/lib/x86_64-linux-gnu/libapr-1.so.0
#1  0x7fb75242ce21 in apr_pool_cleanup_run () from
/usr/lib/x86_64-linux-gnu/libapr-1.so.0
#2  0x0051f268 in stream_pool_cleanup ()
#3  0x7fb7327eb270 in ?? ()
#4  0x7fb730c180a0 in ?? ()
#5  0x7fb7327eb294 in ?? ()
#6  0x7fb7309046c0 in ?? ()
#7  0x7fb730c180a0 in ?? ()
#8  0x30c18028 in ?? ()
#9  0x7fb730c18028 in ?? ()
#10 0x7fb75242b9be in apr_pool_destroy () from
/usr/lib/x86_64-linux-gnu/libapr-1.so.0
#11 0x0051fe61 in h2_stream_destroy ()
#12 0x002d317482d8 in ?? ()
#13 0x7fb730c180a0 in ?? ()
#14 0x7fb7327eb300 in ?? ()
#15 0x00507ab7 in stream_done ()
#16 0x in ?? ()


and or this:
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x7fd2e9034014 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
(gdb) bt
#0  0x7fd2e9034014 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#1  0x7fd2e94c7036 in ?? () from /usr/lib/x86_64-linux-gnu/libapr-1.so.0
#2  0x7fd2e94c746f in apr_hash_set () from
/usr/lib/x86_64-linux-gnu/libapr-1.so.0
#3  0x0052a238 in h2_ihash_remove ()
#4  0x004b in ?? ()
#5  0x7fd2c8d60328 in ?? ()
#6  0x7fd2c97e9300 in ?? ()
#7  0x00506b24 in purge_stream ()
#8  0x7fd2d82140a0 in ?? ()
#9  0x7fd2c831e2d8 in ?? ()
#10 0x7fd20352c0a0 in ?? ()
#11 0x004b in ?? ()
#12 0x7fd2d82140a0 in ?? ()
#13 0x7fd2c831e2d8 in ?? ()
#14 0x7fd2c97e9340 in ?? ()
#15 0x0052a18f in ihash_iter ()
#16 0x7fd2d82140a0 in ?? ()
#17 0x0004 in ?? ()
#18 0x7fd2d82140a0 in ?? ()
#19 0x7fd2c97e93c0 in ?? ()
#20 0x in ?? ()


Greets,
Stefan

Am 17.01.2017 um 21:53 schrieb Stefan Priebe:
> Hi Yann,
> 
> while testing V6 i'm experiencing segfaults.
> 
> exit signal Segmentation
> 
> server-error.log:
> AH00052: child pid 14110 exit signal Segmentation fault (11)
> 
> currently i'm trying to grab a core dump.
> 
> Greets,
> Stefan
> 
> Am 26.12.2016 um 21:18 schrieb Stefan Priebe - Profihost AG:
>> Am 23.12.2016 um 01:41 schrieb Yann Ylavic:
>>> Hi Stefan,
>>>
>>> On Tue, Dec 20, 2016 at 1:52 PM, Stefan Priebe - Profihost AG
>>>  wrote:

 Today i had another server giving no answers to any requests. apache
 fullstatus did not respond.
>>>
>>> Since v5 of the patch, I committed another related change in trunk,
>>> namely: http://svn.apache.org/r1774538
>>> It's about lingering keepalive connections on graceful restart which
>>> may not cause a wakeup.
>>> Does it help?
>>
>> I'll try that but wanted to rebuild based on http 2.4.25. But your
>> mpm_event_listener_wakeup_bug57399_V5 patch does no longer apply to http
>> 2.2.25. Can you rebase it?
>>
 gdb bt shows this for all httpd childs:
>>>
>>> These backtraces are the ones of the main thread, probably not the
>>> culprit.
>>> What does "thread apply all bt" say?
>>
>> Will redo / save that output next time.
>>
>> Thanks!
>>
>> Greets,
>> Stefan
>>
>>> Regards,
>>> Yann.
>>>

Re: JSON for mod_status

[Daniel Gruno]

On 01/17/2017 07:33 PM, Jim Jagielski wrote:
> It all depends on what Bill decides regarding mod_bmx and if
> it is something we intent to backport to 2.4.x
> 
> Still not sure on how to *use* BMX, or how other modules
> "hook in" (right now we have several modules hook into
> mod_status so the "how" is well known and documented), so
> I would require some sort of docs in addition to the actual
> code, of course.

Some JFDI in the meantime; https://httpd.apache.org/server-status :)
JSON: http://httpd.apache.org/server-status?view=json_status

With regards,
Daniel.

> 
>> On Jan 17, 2017, at 12:35 PM, Luca Toscano  wrote:
>>
>>
>>
>> 2016-11-30 18:54 GMT+01:00 Jim Jagielski :
>> I'm thinking about adding JSON support to mod_status...
>> the "plain" version output really stinks and lacks parity
>> w/ the info we provide via HTML, and it would be nice
>> to produce a really easily parseable format.
>>
>> Thoughts...?
>>
>> I know it was extensively discussed, but do we have an agreement about a 
>> plan to add this feature?  :)
>>
>> Luca
>>
>>
>>
> 

Re: HTTP/2 frame prioritization not honored

[Stefan Eissing]

> Am 17.01.2017 um 20:52 schrieb Kyriakos Zarifis :
> 
> Hi Stefan,
> 
> Sorry for the delay, I just got back from traveling. I just tried your new 
> patch and indeed it gets rid of the 100ms delay: The server now serves the 
> high priority object only ~5-20ms (did a few runs) after it receives its 
> request, and only sending 5-6 lower-prio frames in between!

Happy to hear that!

> That's is a dramatic improvement compared to what I was observing in the 
> first experiments (~500ms delay), and I think it affects not only this 
> scenario that I was testing, but any scenario where objects of different 
> priorities are conflicting. To verify this, I also tested another simple 
> scenario in which I aggressively Server-Push several big objects when the 
> server gets the base HTML file. Without the patch, objects embedded in the 
> HTML (requests normally) are backed behind a large fraction of the pushed 
> payload and delayed considerably (500ms). With the patch this is avoided 
> (embedded objects server within a few ms after their request arrives, 
> preempting Pushed objects) 
> 
> If you are interested, I have logs comparing the v1.8.8 performance to the 
> baseline, for both the scenarios ( 1: "prefetched" objects triggered at the 
> end of a page load delaying normal objects from the next navigation, and 2: 
> "server-pushed" objects conflicting with embedded objects on the current page)

Very interested. I'd like to add a page over at https://icing.github.io/mod_h2/ 
about it, so that people can easily grasp what the advantages are. For that, 
your numbers (do you have screenshots of browser timelines maybe?) would be 
very welcome. Also that someone besides the module author has measured it adds 
credibility. :-)

If you write yourself somewhere about it, I am happy to link that.

> Would this patch eventually make it upstream? I'd be very interested in some 
> details on what was causing this and how you resolved it. 

This version sits in the 2.4.x maintenance branch and will be part of the next 
server release. We target releases every quarter and the last was shortly, so 
it might take a while.

The details about what happend was:
 - the callback writing out on the client connection needs to tell the nghttp2 
library explicitly to stop otherwise it will be called over and over again, so 
long as DATA is available.
 - once the write-out stops, mod_http2 is checking if new responses became 
available or if streams need to be resumed. It also checks if there are new 
frames from the client to read. So, basically there are three events that need 
to be served/checked. I have ideas to make that more event triggered, but it is 
basically polling now with timed waits inbetween if nothing is happening.

The change I did first was to never write more than 100ms and then check the 
other things again. That is what you saw in v1.8.7.
In v1.8.8 there is a new atomic flag that signals new response data becoming 
available. This is checked every time before a new frame is assembled for 
writing out. Once the writeout itself is called, there is no more interruption 
at the moment until it is done. That explains the 5-20ms delay you observe now.

Ideally, this all would run poll triggered and I have some ideas how to do 
that. Now I need to find the time to do it.

Nevertheless, thanks to your investigation, we have now a much nicer behaviour 
regarding this and a more responsive server!

Cheers,

Stefan
 

> On Fri, Jan 13, 2017 at 8:43 AM, Stefan Eissing 
>  wrote:
> Hi Kyriakos,
> 
> maybe you can give https://github.com/icing/mod_h2/releases/tag/v1.8.8 a try 
> in your setup? I would be interested if it gets rid of the 100ms delay in 
> response processing. Thanks!
> 
> Cheers,
> 
> Stefan
> 
> > Am 04.01.2017 um 19:27 schrieb Kyriakos Zarifis :
> >
> > Hi Stefan,
> >
> > Yes, this is making a big, obvservable difference!
> >
> > Specifically, in all 3 repeats, the high priority stream is now served 
> > 100ms after it was received, writing ~100 frames (~1.6MB) of currently 
> > served, lower-priority stream.   (was: 500ms, 500frames(~7.5MB))
> >
> > In more detail, after the high-prio request is received, 20 more low-prio 
> > frames are served before the h2_task for it logs that it opens the output 
> > for the new stream. Then, another 80 low-frames are served before the 
> > high-prio reply is written. (relevant logs below)
> >
> > This already has an observable impact on the transition to the next page 
> > the moment I click on the link (goes from 1.5sec to less than 500ms), which 
> > I think is great because this tweak is relevant not just to this scenario, 
> > but to any higher level stream that begins while lower ones are served, 
> > even within a single page.
> >
> > I'm wondering if the change you made can be pushed harder to make the 
> > switch to the new stream even faster, e.g. avoiding even those 100 frames?