Re: svn commit: r1774654 - /httpd/httpd/tags/2.2.25/2.4.x/

2016-12-20 Thread Jacob Champion 

On 12/16/2016 11:23 AM, j...@apache.org wrote:

Removed:
httpd/httpd/tags/2.2.25/2.4.x/


FYI, the git mirror (or, at the very least, the GitHub mirror of our git 
mirror) still has the bogus tag.


--Jacob

HoL blocking of critical resources in HTTP/2

2016-12-20 Thread Kyriakos Zarifis 
Hi,

TL;DR:

I've been experimenting with HTTP/2 Push and Prefetch to preposition
content on the client before it requests it, and specifically I want to
evaluate the pitfalls of prepositioning objects aggressively, with regards
to how it could impact the delivery of higher priority objects.

While experimenting with post-onLoad prepositioning mechanisms to optimize
next page navigations (mostly with prefetch/prerender), I noticed that
pending prepositioned objects can considerably delay the next page; if the
next page navigation is triggered while prepositioned objects are in
flight, the next page HTML will be backed behind a considerable part of
them on the server.

>From what I can tell, object/stream priorities cannot help since the HoL
blocking happens at the SSL socket where the server can't preempt data that
it has already written.

So I wonder if there could be some logic on the server that controls how
aggressively it writes low-priority frames to the SSL socket, making sure
they are being drained before pushing more and bloating the buffer (and
causing critical objects that are requested soon after to be delayed).



Details:

To evaluate the impact, I tried a simple, extreme scenario where I trigger
the Prefetch (low-priority) of a large image after the onLoad of page A
fires, and soon after it fires and the image is being downloaded I manually
navigate to page B. In this case it takes as long as it takes for the image
to download before the next page starts showing. That was many seconds in
my case since it was a purposefully large (5MB) image and bad connection
(4G, ~200ms from server).

I also tried prefetching 8 copies of the same image, over a fast connection
(cable, 20ms to the server). I navigate to page B while the first few
prefetched images are being downloaded. In this case page B's HTML arrived
900ms after the request was sent. Looking at Chrome's event log, the HTML
(stream weight 256) was indeed prioritized over the prefetched images
(weight 110): the HTML arrives in the middle of the 4th prefetched image -
but there was still more than 1xBDP worth of data that arrived during the
900ms between the HTML request-response.

I believe that these are H2 frames that have been written for sending (but
not yet sent on the wire), where the server can no longer preempt them so
anything new (regardless of priority) can't be sent on the wire before the
buffer is drained.

My hypothesis was that priorities are honored at the application level -
i.e. the HTML frames are interleaved into the socket as soon as the request
arrives (I think I've now verified that from both server and chrome logs) -
but the socket doesn't keep up at draining the previously written low-prio
frames. I've also observed that the HTML's delay is in multiples of RTTs.
So I guess I'm just describing an instance of bufferbloat, which is likely
to happen when many/large low-prio requests arrive at the server before a
critical one (as is the case with prepositioning)

So I think some Head-of-LIne blocking (which H2 is supposed to do away
with) is happening even though priorities are honored in the application
layer. As a result the client receives the critical object (much) later
than it could have.

I considered tweaking the browser to open a second connection for
prefetch/pushed/otherwise low-prio content, but this goes against the
goals/purpose of HTTP/2. I believe that the right place to tweak this would
be the server. I think the application does things right as far as it
knows, i.e. frames leave the application layer in OK priority, but not the
wire.

What do you think would be the right way to fix this (assuming it is indeed
something that needs fixing)? Do you think there could be some additional
logic that controls how aggressively the server writes to the SSL socket,
that checks how much of low-prio data has been drained from the socket
before writing more, to avoid bloating the buffer on the server side with
low-priority frames which more critical frames will have to wait behind
when they become available?

I've started looking at httpd's source code to get a sense of how big of a
change this would be, but since I haven't looked at the code before I
wanted to get your thoughts on whether this makes sense first.

Thanks and sorry for the length

Re: svn commit: r17503 - in /dev/httpd: Announcement2.4.html Announcement2.4.txt

2016-12-20 Thread William A Rowe Jr 
On Tue, Dec 20, 2016 at 12:34 PM, William A Rowe Jr 
wrote:

> On Tue, Dec 20, 2016 at 12:13 PM,  wrote:
>
>> Author: wrowe
>> Date: Tue Dec 20 18:13:12 2016
>> New Revision: 17503
>>
>> Log:
>> Record security errata, edits in the next 45 minutes are most welcomed
>>
>> Modified:
>> dev/httpd/Announcement2.4.html
>> dev/httpd/Announcement2.4.txt
>>
>
> I'll take the Announcement live on the hour (30 min from now), anyone
> who wants to edit, the path is a little unusual;
>
> https://dist.apache.org/repos/dist/dev/httpd
>
> I'll pick up any final edits before replicating to release/dev/httpd
>
> Trying to get as much of httpd_vulnerabilities.xml caught up ahead
> of the announce as is reasonable. Doesn't help that the latest
> dnf update vim completely smashed my vim behavior ;-/
>

Accomplished with some help from jchampion as I didn't have any
smtp set up for @apache personality (the joys of being gmail-bound.)

Review of the website etc welcomed. Thanks to all who helped get
this ready, thanks again to Jim for RM'ing!

Re: svn commit: r1775186 - /httpd/test/framework/trunk/t/apache/http_strict.t

2016-12-20 Thread Rainer Jung 

Hi Bill,

Am 20.12.2016 um 18:22 schrieb William A Rowe Jr:

On Mon, Dec 19, 2016 at 4:20 PM, > wrote:

Author: rjung
Date: Mon Dec 19 22:20:12 2016
New Revision: 1775186

URL: http://svn.apache.org/viewvc?rev=1775186=rev

Log:
Skip tests that need a cgi module if cgi
is not available.

Modified:
httpd/test/framework/trunk/t/apache/http_strict.t

Modified: httpd/test/framework/trunk/t/apache/http_strict.t
URL:

http://svn.apache.org/viewvc/httpd/test/framework/trunk/t/apache/http_strict.t?rev=1775186=1775185=1775186=diff



==
--- httpd/test/framework/trunk/t/apache/http_strict.t (original)
+++ httpd/test/framework/trunk/t/apache/http_strict.t Mon Dec 19
22:20:12 2016
@@ -122,6 +122,10 @@ foreach my $t (@test_cases) {
 my $decoded;

 if ($req =~ s/^R//) {
+if (!have_cgi) {
+skip "Skipping test without CGI module";
+next;
+}


Does this work in the presence of either cgi and/or cgid?


Yes, have_cgi is true if either one of the two is loaded.

Regards,

Rainer

Re: Old Announcement Logo

2016-12-20 Thread Jacob Champion 

On 12/20/2016 11:39 AM, William A Rowe Jr wrote:

Sure thing. I only needed ...2.4.txt in its final form. Go for it :)

Bill


I'll hold off until next time, so we don't have too many moving parts 
for this release.


--Jacob

Re: Old Announcement Logo

2016-12-20 Thread William A Rowe Jr 
Sure thing. I only needed ...2.4.txt in its final form. Go for it :)

Bill

On Tue, Dec 20, 2016 at 1:02 PM, Jacob Champion 
wrote:

> Rich pointed out that our announcements are still using the old logo:
>
> https://www.apache.org/dist/httpd/Announcement2.4.html
>
> Should we add our new banner (https://httpd.apache.org/imag
> es/httpd_logo_wide_new.png) to the dist repo and use it instead?
>
> --Jacob
>

Old Announcement Logo

2016-12-20 Thread Jacob Champion 

Rich pointed out that our announcements are still using the old logo:

https://www.apache.org/dist/httpd/Announcement2.4.html

Should we add our new banner 
(https://httpd.apache.org/images/httpd_logo_wide_new.png) to the dist 
repo and use it instead?


--Jacob

Re: svn commit: r17503 - in /dev/httpd: Announcement2.4.html Announcement2.4.txt

2016-12-20 Thread William A Rowe Jr 
On Tue, Dec 20, 2016 at 12:13 PM,  wrote:

> Author: wrowe
> Date: Tue Dec 20 18:13:12 2016
> New Revision: 17503
>
> Log:
> Record security errata, edits in the next 45 minutes are most welcomed
>
> Modified:
> dev/httpd/Announcement2.4.html
> dev/httpd/Announcement2.4.txt
>

I'll take the Announcement live on the hour (30 min from now), anyone
who wants to edit, the path is a little unusual;

https://dist.apache.org/repos/dist/dev/httpd

I'll pick up any final edits before replicating to release/dev/httpd

Trying to get as much of httpd_vulnerabilities.xml caught up ahead
of the announce as is reasonable. Doesn't help that the latest
dnf update vim completely smashed my vim behavior ;-/

Re: [RESULT] (Was: Re: [VOTE] Release Apache httpd 2.4.25 as GA)

2016-12-20 Thread William A Rowe Jr 
On Mon, Dec 19, 2016 at 12:32 PM, Jim Jagielski  wrote:

> With 72 hours up, and sufficient +1 (binding) votes, I call
> this vote CLOSED with the result of PASS.
>
> I have gone ahead and moved the release artifacts over the
> the required place for the mirrors to sync up.
>

Thanks for RM'ing!


> I will be away (In Tyson's Corner, VA) later today and
> all day tomorrow, and the 1st half of Weds), so if someone
> wants to take on the announcing part, that would be great.
> I'll have limited cycles and external net access.
>

I'm on it.

Re: T of 2.4.24

2016-12-20 Thread William A Rowe Jr 
On Fri, Dec 9, 2016 at 1:44 AM, Yann Ylavic  wrote:

> On Thu, Dec 8, 2016 at 7:16 PM, William A Rowe Jr 
> wrote:
> >
> > It does raise the question again of whether the httpd project can
> distribute
> > a source code package on www.apache.org/dist/httpd/ which is not voted
> > on by the project, and whether it violates the spirit of the pmc
> consensus
> > to no longer be the distributor of dependencies which frequently fall
> into
> > a poorly maintained/updated state.
>
> Current httpd-2.4.23-deps.tar.*/srclib seem to contain APR(-util)
> only, no expat or PCRE, wasn't this decision taken already?
>

The decision in Nov 2008 to drop pcre was followed, that was not in
any -deps 'not-a-release' tarball. Expat was more deeply embedded;

httpd-2.4.23/srclib/apr-util/xml/expat/


> httpd-2.2.32.tar.*/srclib contain PCRE 5.0 (according to Changelog),
> no expat, but it looks off topic for this T
>

Yup, I'm working on language that would accompany httpd-2.2.32.tar.gz
that the distribution includes ancient, bundled legacy binary-compatible
pcre and expat, and that users are strongly cautioned to provision pcre,
expat and the most current versions of apr and apr-util themselves from
the respective projects or their OS distribution.

Re: Better ./configure --help wording for paths?

2016-12-20 Thread William A Rowe Jr 
My first attempt at this was a major change to how --with-pcre= now
works on trunk, it is now a path to .../pcre[2]-config,
.../bin/pcre[2]-config,
(first it tries the {target}-pcre[2]-config for cross-compilation, and then
the undecorated flavor.)

I did hack around providing either a full path including the script name,
or only the config script name (e.g. --with-pcre=pcre-config to revert to
a legacy pcre-8.x). That causes some extra hassle but seems like a
good idea for these edge cases. In these cases, or where --with-pcre
is simply not given, we actually walk the $PATH, and --with-pcre=PATH
can actual be a traditional path list in place of default $PATH.

So it seems --with-pcre=PATH is the right textual description for this
specific case, and changing these without potentially providing a list
of paths would be erroneous.

I'd welcome any feedback and review of trunk's --with-pcre= detection.

Bill



On Wed, Nov 16, 2016 at 6:40 AM, William A Rowe Jr 
wrote:

> Can we agree on a keyword/wording convention here for httpd-2.5-dev?
>
>   --with-apr=PATH prefix for installed APR or the full path to
>   --with-pcre=PATHUse external PCRE library
>   --with-valgrind[=DIR]   Enable code to reduce valgrind false positives
>   (optionally: set path to valgrind headers)
>   --with-distcache=PATH   Distcache installation directory
>   --with-z=PATH   use a specific zlib library
>   --with-libxml2=DIR  Specify location of libxml2
>   --with-brotli=PATH  Brotli installation directory
>   --with-lua=PATH Path to the Lua 5.2/5.1 prefix
>   --with-serf=PATHSerf client library
>   --with-ssl=PATH OpenSSL installation directory
>   --with-nghttp2=PATH nghttp2 installation directory
>   directory.
>
> Contrast to apr-2.0 and the autoconf keywords and text;
>
> Installation directories:
>   --prefix=PREFIX install architecture-independent files in PREFIX
>   [/usr/local/apr]
>   --exec-prefix=EPREFIX   install architecture-dependent files in EPREFIX
>   [PREFIX]
>
> By default, `make install' will install all the files in
> `/usr/local/apr/bin', `/usr/local/apr/lib' etc.  You can specify
> an installation prefix other than `/usr/local/apr' using `--prefix',
> for instance `--prefix=$HOME'.
>
> For better control, use the options below.
>
> Fine tuning of the installation directories:
>   --bindir=DIRuser executables [EPREFIX/bin]
>   --sbindir=DIR   system admin executables [EPREFIX/sbin]
>   --libexecdir=DIRprogram executables [EPREFIX/libexec]
>   --sysconfdir=DIRread-only single-machine data [PREFIX/etc]
>   --sharedstatedir=DIRmodifiable architecture-independent data
> [PREFIX/com]
>   --localstatedir=DIR modifiable single-machine data [PREFIX/var]
>   --libdir=DIRobject code libraries [EPREFIX/lib]
>   --includedir=DIRC header files [PREFIX/include]
>   --oldincludedir=DIR C header files for non-gcc [/usr/include]
>   --datarootdir=DIR   read-only arch.-independent data root
> [PREFIX/share]
>   --datadir=DIR   read-only architecture-independent data
> [DATAROOTDIR]
>   --infodir=DIR   info documentation [DATAROOTDIR/info]
>   --localedir=DIR locale-dependent data [DATAROOTDIR/locale]
>   --mandir=DIRman documentation [DATAROOTDIR/man]
>   --docdir=DIRdocumentation root [DATAROOTDIR/doc/PACKAGE]
>   --htmldir=DIR   html documentation [DOCDIR]
>   --dvidir=DIRdvi documentation [DOCDIR]
>   --pdfdir=DIRpdf documentation [DOCDIR]
>   --psdir=DIR ps documentation [DOCDIR]
>   --with-sysroot=DIR Search for dependent libraries within DIR
>   --with-installbuilddir=DIR location to store APR build files (defaults
> to '${datadir}/build')
>   --with-efence[=DIR] path to Electric Fence installation
>   --with-valgrind[=DIR]   Enable code to teach valgrind about apr pools
>   (optionally: set path to valgrind headers)
>   --with-egd[=DIR]use EGD-compatible socket
>   --with-openssl=DIR  specify location of OpenSSL
>   --with-nss=DIR  specify location of NSS
>   --with-commoncrypto=DIR specify location of CommonCrypto
>   --with-gdbm=DIR enable GDBM support
>   --with-ndbm=PATHFind the NDBM header and library in
> `PATH/include'
>   and `PATH/lib'. If PATH is of the form
> `HEADER:LIB',
>   library in LIB. If you omit the `=PATH' part
>   --with-berkeley-db=PATH Find the Berkeley DB header and library in
>   `PATH/include' and `PATH/lib'. If PATH is of the
>   `=PATH' part completely, the configure script
> will
>   --with-pgsql=DIRspecify PostgreSQL location
>   --with-mysql=DIRenable MySQL DBD driver
>

Re: svn commit: r1775186 - /httpd/test/framework/trunk/t/apache/http_strict.t

2016-12-20 Thread William A Rowe Jr 
On Mon, Dec 19, 2016 at 4:20 PM,  wrote:

> Author: rjung
> Date: Mon Dec 19 22:20:12 2016
> New Revision: 1775186
>
> URL: http://svn.apache.org/viewvc?rev=1775186=rev
> Log:
> Skip tests that need a cgi module if cgi
> is not available.
>
> Modified:
> httpd/test/framework/trunk/t/apache/http_strict.t
>
> Modified: httpd/test/framework/trunk/t/apache/http_strict.t
> URL: http://svn.apache.org/viewvc/httpd/test/framework/trunk/t/
> apache/http_strict.t?rev=1775186=1775185=1775186=diff
> 
> ==
> --- httpd/test/framework/trunk/t/apache/http_strict.t (original)
> +++ httpd/test/framework/trunk/t/apache/http_strict.t Mon Dec 19 22:20:12
> 2016
> @@ -122,6 +122,10 @@ foreach my $t (@test_cases) {
>  my $decoded;
>
>  if ($req =~ s/^R//) {
> +if (!have_cgi) {
> +skip "Skipping test without CGI module";
> +next;
> +}
>

Does this work in the presence of either cgi and/or cgid?

Re: Frequent wake-ups for mpm_event

2016-12-20 Thread Stefan Priebe - Profihost AG 
Hi Yann,

here is a follow up on this old thread with some problems again. See below.

Am 23.09.2016 um 21:26 schrieb Stefan Priebe - Profihost AG:
> Hi Yann,
>   Hi Luca,
> 
> right now i had another server which shows:
> AH00485: scoreboard is full, not at MaxRequestWorkers msg.
> 
> I never saw that before applying the event patch.
> 
> A fullstatus view is not answered - so the whole apache hangs. Also it
> has 84 running processes instead of normally just 4-8.
> # ps aux|grep httpd | wc -l
> 84
> 
> There are also not many connections:
> # netstat -na | egrep ":80|:443"|wc -l
> 110
> 
> Even after a restart all and even more httpd processes are there:
> # ps aux|grep httpd | wc -l
> 89
> 
> ps aux also tell me that there a lot of pretty old processes:
> # ps aux|grep httpd | grep Sep22 | wc -l
> 40
> 
> and even a lot of processes are from Sep 23 many hours ago. So i don't
> think this is bug #53555.
> 
> It just seems a lot of threads are hanging / deadlocked.
> 
> gdb says:
> (gdb) bt
> #0  0x7f9235b634db in pthread_join () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #1  0x7f9235d9e9bb in apr_thread_join () from
> /usr/lib/x86_64-linux-gnu/libapr-1.so.0
> #2  0x0055b283 in join_workers ()
> #3  0x0055b8cc in child_main ()
> #4  0x0055ba3f in make_child ()
> #5  0x0055c21e in perform_idle_server_maintenance ()
> #6  0x0055c619 in server_main_loop ()
> #7  0x0055c959 in event_run ()
> #8  0x00445f91 in ap_run_mpm ()
> #9  0x0043dd19 in main ()
> 
> 
> bt from all threads of an old httpd process:
> [Thread debugging using libthread_db enabled]
> Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
> 0x7f9235b634db in pthread_join () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> 
> Thread 52 (Thread 0x7f92325ee700 (LWP 13704)):
> #0  0x7f9235b6608f in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #1  0x005286f9 in get_mplx_next ()
> #2  0x0052fb34 in execute ()
> #3  0x7f9235b620a4 in start_thread () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #4  0x7f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6
> 
> Thread 51 (Thread 0x7f9231ded700 (LWP 13705)):
> #0  0x7f9235b6608f in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #1  0x005286f9 in get_mplx_next ()
> #2  0x0052fb34 in execute ()
> #3  0x7f9235b620a4 in start_thread () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #4  0x7f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6
> 
> Thread 50 (Thread 0x7f92315ec700 (LWP 13706)):
> #0  0x7f9235b6608f in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #1  0x005286f9 in get_mplx_next ()
> #2  0x0052fb34 in execute ()
> #3  0x7f9235b620a4 in start_thread () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #4  0x7f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6
> 
> Thread 49 (Thread 0x7f9230deb700 (LWP 13707)):
> #0  0x7f9235b6608f in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #1  0x005286f9 in get_mplx_next ()
> #2  0x0052fb34 in execute ()
> #3  0x7f9235b620a4 in start_thread () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #4  0x7f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6
> 
> Thread 48 (Thread 0x7f92305ea700 (LWP 13708)):
> #0  0x7f9235b6608f in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #1  0x005286f9 in get_mplx_next ()
> #2  0x0052fb34 in execute ()
> #3  0x7f9235b620a4 in start_thread () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #4  0x7f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6
> 
> Thread 47 (Thread 0x7f922fde9700 (LWP 13709)):
> #0  0x7f9235b6608f in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #1  0x005286f9 in get_mplx_next ()
> #2  0x0052fb34 in execute ()
> #3  0x7f9235b620a4 in start_thread () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #4  0x7f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6
> 
> Thread 46 (Thread 0x7f922f5e8700 (LWP 13710)):
> #0  0x7f9235b6608f in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #1  0x005286f9 in get_mplx_next ()
> #2  0x0052fb34 in execute ()
> #3  0x7f9235b620a4 in start_thread () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #4  0x7f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6
> 
> Thread 45 (Thread 0x7f922ede7700 (LWP 13711)):
> #0  0x7f9235b6608f in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #1  0x005286f9 in get_mplx_next ()
> #2  0x0052fb34 in execute ()
> #3  0x7f9235b620a4 in start_thread () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #4