Re: [VOTE] Release Apache httpd 2.4.28 as GA

2017-09-28 Thread Rainer Jung 
An update concerning the sporadic proxy test failures for prefork on 
Solaris that I observed:


These are a bit difficult to nail down, because running only the proxy 
tests does not fail. One has to run a relatively big part of the test 
suite to have a chance of triggering the problem during the proxy test. 
I can reproduce in about 50% of the test runs.


Currently I have the impression, that the problem only happens with apr 
1.6.x (in about 50% of the test runs), but never with apr 1.5.x. Whether 
I use apu 1.5.x or apu 1.6.x does not seem to matter.


So I need to dig deeper, what behavioral changes are expected between 
apr 1.5.x and 1.6.x especially on Solaris (and which only matter for 
prefork).


Still IMHO not a show-stopper but of course worth fixing.

Regards,

Rainer

Re: [VOTE] Release Apache httpd 2.4.28 as GA

2017-09-28 Thread William A Rowe Jr 
On Mon, Sep 25, 2017 at 12:11 PM, Steffen  wrote:
> On Windows it does not build out of the box.
>
> Missing modules/core include for mod_watchdog.h in
> mod_proxy_balancer.dsp/mak and libhttp.dsp/mak . Did not checked cmake.

Seems baffling, but it is pretty straightforward in hindsight.

libhttpd is built first before modules/*

libhttpd compiles in mod_so (alongside core, win32, mpm_winnt, and http.)

libhttpd starts by copying mod_so.h to include/, originally the only module,
just as the unix build had long done so when mod_so was enabled.
There was no need to /I modules/core with that header in include/

modules/core/mod_watchdog was added - but built as a loadable module.
The .h was neither copied to include/ nor was /I include/core added.

mod_heartbeat, mod_heartmonitor, mod_proxy_balancer and mod_proxy_hcheck
all include mod_watchdog; it makes more sense to copy this header as well,
rather than modify four projects. I'm a little confused how libhttpd would need
it itself, but that's where include copying should occur for now.

Fixed in r1810012

Re: [VOTE] Release Apache httpd 2.4.28 as GA

2017-09-28 Thread Helmut K. C. Tessarek 
On 2017-09-28 09:51, Jim Jagielski wrote:
> Personally, I don't think the age or heritage of the build-logic is the issue,
> but rather a lack of people *really* testing 2.4.x until a release is tagged.
> It's for that reason that I tend to pre-announce a T well in advance of 
> actually
> DOING the T so that people can test HEAD in hopes that the tag will already
> have had some good testing beforehand.

I have a question. Why are you tagging a release and do testing? Most of
the time a problem is found and a new release is tagged and it starts
over (I think the max was a 3 or 4 patch level jump).

Why not tagging an RC? People test the RC. When all is ok, the RC is
released. If not a new RC is tagged.

Cheers,
 K. C.

-- 
regards Helmut K. C. Tessarek  KeyID 0xF7832007C11F128D
Key fingerprint = 28A3 1666 4FE8 D72C CFD5 8B23 F783 2007 C11F 128D

/*
   Thou shalt not follow the NULL pointer for chaos and madness
   await thee at its end.
*/

Re: [VOTE] Release Apache httpd 2.4.28 as GA

2017-09-28 Thread Jim Jagielski 
If the issue is "real" and a patch is forthcoming, I agree that we
mark 2.4.28 as DOA and proceed w/ 2.4.29 next week. I am comfortable keeping
the VOTE open longer that the normal/required 72hours to be sure one way
or another.

Personally, I don't think the age or heritage of the build-logic is the issue,
but rather a lack of people *really* testing 2.4.x until a release is tagged.
It's for that reason that I tend to pre-announce a T well in advance of 
actually
DOING the T so that people can test HEAD in hopes that the tag will already
have had some good testing beforehand.

> On Sep 27, 2017, at 11:21 PM, William A Rowe Jr  wrote:
> 
> The assert() has me concerned, and Steffen's report is problematic. He has a 
> vote but hasn't cast it. At this moment I'm -0 and would spin a 2.4.29 next 
> week to address these issues, unless you decide to respin before this 
> release, yourself.
> 
> Nothing I've changed today altered the httpd tarball significantly. Studying 
> Steffen's report next, along with some apparently missing glue for brotli.
> 
> My solution is going to be radical, shove every last d*mned modules/foo into 
> the /I path includes list so this can't happen again during 2.4, and 
> hopefully not until 20 year old build logic is discarded. One less thing to 
> worry about or pre-review when RM's loudly announce an upcoming tag.
> 
> 
> 
> 
> On Sep 25, 2017 07:13, "Jim Jagielski"  wrote:
> The pre-release test tarballs for Apache httpd
> version 2.4.28 can be found at the usual place:
> 
> http://httpd.apache.org/dev/dist/
> 
> I'm calling a VOTE on releasing these as Apache httpd 2.4.28 GA.
> 
> [ ] +1: Good to go
> [ ] +0: meh
> [ ] -1: Danger Will Robinson. And why.
> 
> Vote will last the normal 72 hrs.
> 
> NOTE: The *-deps are only there for convenience.
> 
> Thx!

Re: [VOTE] Release Apache httpd 2.4.28 as GA

2017-09-28 Thread Rainer Jung 

Am 25.09.2017 um 14:13 schrieb Jim Jagielski:

The pre-release test tarballs for Apache httpd
version 2.4.28 can be found at the usual place:

http://httpd.apache.org/dev/dist/

I'm calling a VOTE on releasing these as Apache httpd 2.4.28 GA.

[X] +1: Good to go
[ ] +0: meh
[ ] -1: Danger Will Robinson. And why.

Vote will last the normal 72 hrs.

NOTE: The *-deps are only there for convenience.


+1 to release and thanks a bunch for RMing and to all who contributed to 
the release.


In short: No regressions found.

Detailed report:

- Sigs and hashes OK
- contents of tarballs identical
- contents of tag and tarballs identical
  except for expected deltas

Built on

- Solaris 10 Sparc as 32 Bit Binaries
- SLES 11+12 (64 Bits)
- RHEL 6+7 (64 Bits)

For all platforms built

- with default (shared), shared and static modules
- with module sets none, few, most, all, reallyall and default
  (always mod_privileges disabled)
- using --enable-load-all-modules
- against "included" APR/APU from deps tarball,
  plus external APR/APU 1.6.2/1.6.0, 1.5.2/1.5.4,
  1.6.x branches head and trunk head.

- using external libraries
  - expat 2.2.4
  - pcre 8.41
  - openssl 1.0.2k plus patches
  - lua 5.3.4 (compiled with LUA_COMPAT_MODULE)
  - distcache 1.5.1
  - libxml2 2.9.5
  - libnghttp2 1.26.0
  - brotli 1.0.1

- Tool chain:
- platform gcc except on Solaris
  (gcc 4.1.2 for Solaris 8 and 7.1.0 Solaris 10 (only older APR/APU 
1.5.x compiled with older gcc 4.9.2)

- CFLAGS: -O2 -g -Wall -fno-strict-aliasing
  - on Solaris additionally -mpcu=v9, -D_XOPEN_SOURCE,
-D_XOPEN_SOURCE_EXTENDED=1 and -D__EXTENSIONS__
plus -D_XPG6 when using gcc 7.1.0

350 build succeeded, 2 builds both on SLES 11 crashed due to 
segmentation faults in ksh during make, but could be completed by 
restarting make. Not a regression.


- compiler warnings:

  - modules/core/mod_watchdog.c:436: warning: 'rv' may be used
uninitialized in this function
  -> warning is correct but nt critical (debug log), will fix this soon;
 not a regression

  and only on Solaris (gcc 7.1.0)

  - server/util.c:2667:10: warning: 'escaped_char[0]' may be used
uninitialized in this function [-Wmaybe-uninitialized]

  - srclib/apr-util/xlate/xlate.c:120:38: warning: passing argument 2 of
'iconv' from incompatible pointer type
[-Wincompatible-pointer-types]

  - srclib/apr-util/xlate/xlate.c:343:42: warning: passing argument 2 of
'iconv' from incompatible pointer type
[-Wincompatible-pointer-types]


Tested for

- Solaris 10, SLES 11+12, RHEL 6+7
- MPMs prefork, worker, event
- default, shared and static modules
- log level trace8
- module set reallyall (125 modules plus MPMs)

The following test failures were seen:

a Lots of tests in t/module/session.t fail always for static builds
  The whole setup for the /sesiontest uri is missing in the generated
  t/conf/httpd.conf. This is due to it missing from the also generated
  filet/conf/apache_test_config.pm. I do not know yet, why it is missing
  there, but this seems to be a test framework problem.
  Not a regression.

b Test 5 in t/modules/dav.t:
  Happens for 7 out of about 225 runs
  Creation, modified and now times not in the correct order.
  This seems to be a system issue, all tests done on NFS,
  many tested on virtualized guests.
  Not a regression.

c Test 59 of t/modules/include.t only and always on
  Solaris.
  This is due to a bug in the test, which uses strftime()
  with a "%s" pattern that is not supported on Solaris.
  Until recently the server and the test client both returned
  verbatim "%s" and the test succeeded. After updating some
  Perl modules for the http2 tests, the perl client even
  on Solaris now supports "%s" in strftime and the test starts
  to fail. It seems we have to fix the test.
  Not a regression

d Various tests in t/apache/expr_string.t: (3, 6, 11, 14, 17, 20 ,23)
  Happens for 8 out of about 225 runs (this time all on RHEL6).
  The failure is always on line 68, where the error_log contents
  are checked.
  Not a regression.

e Only on Solaris and only with prefork proxy tests sometimes
  seem to hang until timeout. Some test runs complete without
  the issue. Once the issue hits, all remaining proxy tests during
  that run face that issue.
  Not observed for static builds. Of the 8 non-static builds using
  APR/APU 1.6 or trunk, 5 had the problem. The two APR/APU builds did
  not have it. First observed when testing 2.4.26.
  I'll try to investigate further, but IMHO this is not a show-stopper.

All in all I don't see a critical problem.

Regards,

Rainer

Re: [VOTE] Release Apache httpd 2.4.28 as GA

2017-09-28 Thread Stefan Eissing 
Update: disregard the man behind the curtain - for now.

I have the strangest effects on my main machine under native macOS 10.12.6, 
which
do not happen on my parallels ubuntu image and my laptop with macOS 10.13.0. I 
tested
with hyperthreading disabled to exclude the Intel *lake bugs, but it made no 
difference.

I will upgrade the machine over night to macOS 10.13.0 and check again tomorrow 
if
that had any effect.

Regardless of all this, I managed to find an edge case assertion fail and made 
a fix
on trunk. If we go for 2.4.29, I'll propose that for inclusion. It is not a 
regression to
2.4.27, so no veto from me on 2.4.28.

Cheers,

Stefan


> Am 28.09.2017 um 09:54 schrieb Stefan Eissing :
> 
> Analyzing...
> 
> my checkout of branch/2.4.x does not show this.
> my buildt of 2.4.28 with -deps as provided, does after while of letting:
> 
> while true; do h2load -t 8 -c 100 -n 1 -m 5 -d gen/data-10k 
> http://test.example.org:12345/; done
> 
> run (data-10k is a 10k text file). The whole server setup is according to the 
> mod_http2 test cases
> in  https://svn.apache.org/repos/asf/httpd/test/mod_h2/trunk
> 
> In 2.4.28 I have apr 1.6.2 and in my branch/2.4.x is a 1.5.x (which shows 
> 1.5.3 in its header).
> 
> Will check if that is connected or not. Will also check if this happens for 
> me on my Ubuntu.
> 
> Sporadically (less frequent), I can also see (not both 2.4.x and 2.4.28):
> Thread 3 Crashed:
> 0   libsystem_kernel.dylib0x7fffd3c34d42 __pthread_kill + 10
> 1   libsystem_pthread.dylib   0x7fffd3d22457 pthread_kill + 90
> 2   libsystem_c.dylib 0x7fffd3b9a4bb __abort + 140
> 3   libsystem_c.dylib 0x7fffd3b9a42f abort + 144
> 4   libsystem_pthread.dylib   0x7fffd3d23bc7 __pthread_abort + 49
> 5   libsystem_pthread.dylib   0x7fffd3d23c7b 
> __pthread_abort_reason + 180
> 6   libsystem_pthread.dylib   0x7fffd3d1fd93 
> _pthread_mutex_unlock_drop + 167
> 7   mod_http2.so  0x00010cb42b2b h2_beam_send + 1819 
> (h2_bucket_beam.c:965)
> 8   mod_http2.so  0x00010cb59d31 send_out + 257 
> (h2_task.c:100)
> 9   mod_http2.so  0x00010cb59146 
> h2_filter_slave_output + 294 (h2_task.c:176)
> 10  httpd 0x00010c598829 
> ap_process_request_after_handler + 89 (http_request.c:366)
> 11  httpd 0x00010c598ab6 ap_process_request + 
> 22 (http_request.c:473)
> 12  mod_http2.so  0x00010cb5874e h2_task_process_conn 
> + 398 (h2_task.c:682)
> 13  httpd 0x00010c5705f7 
> ap_run_process_connection + 55 (connection.c:42)
> 14  mod_http2.so  0x00010cb599db h2_task_do + 539 
> (h2_task.c:640)
> 15  mod_http2.so  0x00010cb5df24 slot_run + 260 
> (h2_workers.c:233)
> 16  libsystem_pthread.dylib   0x7fffd3d1f93b _pthread_body + 180
> 17  libsystem_pthread.dylib   0x7fffd3d1f887 _pthread_start + 286
> 18  libsystem_pthread.dylib   0x7fffd3d1f08d thread_start + 13
> 
> which is a call to apr_thread_mutex_unlock(). Without having checkinto 
> pthread version, the closest
> similar report I could find on the net is 
> https://sourceware.org/bugzilla/show_bug.cgi?id=17514
> 
> Will report when I know more.
> 
> -Stefan
> 
>> Am 27.09.2017 um 18:04 schrieb Luca Toscano :
>> 
>> Hi Stefan,
>> 
>> 2017-09-27 17:32 GMT+02:00 Stefan Eissing :
>> On my h2 load tests, the server sometimes crashes in an assertion failure 
>> (build in maintainer mode):
>> 
>> How did you make the tests? It would be good for people to attempt to 
>> reproduce..
>> 
>> 
>> macOS crash reporter:
>> Thread 54 Crashed:
>> 0   libsystem_kernel.dylib  0x7fffd3c34d42 __pthread_kill + 
>> 10
>> 1   libsystem_pthread.dylib 0x7fffd3d22457 pthread_kill + 90
>> 2   libsystem_c.dylib   0x7fffd3b9a4bb __abort + 140
>> 3   libsystem_c.dylib   0x7fffd3b9a42f abort + 144
>> 4   httpd   0x00010be99282 ap_log_assert + 
>> 130 (log.c:1696)
>> 5   mod_mpm_event.so0x00010c50b51c ap_queue_push + 
>> 188
>> 6   mod_mpm_event.so0x00010c509be2 listener_thread + 
>> 2226 (event.c:1749)
>> 7   libsystem_pthread.dylib 0x7fffd3d1f93b _pthread_body + 
>> 180
>> 8   libsystem_pthread.dylib 0x7fffd3d1f887 _pthread_start + 
>> 286
>> 9   libsystem_pthread.dylib 0x7fffd3d1f08d thread_start + 13
>> 
>> error_log:
>> [Wed Sep 27 17:27:09.941040 2017] [mpm_event:notice] [pid 23404:tid 
>> 140736895005632] AH00489: Apache/2.4.28 (Unix) OpenSSL/1.1.0e configured -- 
>> resuming normal operations
>> [Wed Sep 27 17:27:09.941055 2017]

Re: [VOTE] Release Apache httpd 2.4.28 as GA

2017-09-28 Thread Stefan Eissing 
Analyzing...

my checkout of branch/2.4.x does not show this.
my buildt of 2.4.28 with -deps as provided, does after while of letting:

while true; do h2load -t 8 -c 100 -n 1 -m 5 -d gen/data-10k 
http://test.example.org:12345/; done

run (data-10k is a 10k text file). The whole server setup is according to the 
mod_http2 test cases
in  https://svn.apache.org/repos/asf/httpd/test/mod_h2/trunk

In 2.4.28 I have apr 1.6.2 and in my branch/2.4.x is a 1.5.x (which shows 1.5.3 
in its header).

Will check if that is connected or not. Will also check if this happens for me 
on my Ubuntu.

Sporadically (less frequent), I can also see (not both 2.4.x and 2.4.28):
Thread 3 Crashed:
0   libsystem_kernel.dylib  0x7fffd3c34d42 __pthread_kill + 10
1   libsystem_pthread.dylib 0x7fffd3d22457 pthread_kill + 90
2   libsystem_c.dylib   0x7fffd3b9a4bb __abort + 140
3   libsystem_c.dylib   0x7fffd3b9a42f abort + 144
4   libsystem_pthread.dylib 0x7fffd3d23bc7 __pthread_abort + 49
5   libsystem_pthread.dylib 0x7fffd3d23c7b 
__pthread_abort_reason + 180
6   libsystem_pthread.dylib 0x7fffd3d1fd93 
_pthread_mutex_unlock_drop + 167
7   mod_http2.so0x00010cb42b2b h2_beam_send + 1819 
(h2_bucket_beam.c:965)
8   mod_http2.so0x00010cb59d31 send_out + 257 
(h2_task.c:100)
9   mod_http2.so0x00010cb59146 
h2_filter_slave_output + 294 (h2_task.c:176)
10  httpd   0x00010c598829 
ap_process_request_after_handler + 89 (http_request.c:366)
11  httpd   0x00010c598ab6 ap_process_request + 
22 (http_request.c:473)
12  mod_http2.so0x00010cb5874e h2_task_process_conn 
+ 398 (h2_task.c:682)
13  httpd   0x00010c5705f7 
ap_run_process_connection + 55 (connection.c:42)
14  mod_http2.so0x00010cb599db h2_task_do + 539 
(h2_task.c:640)
15  mod_http2.so0x00010cb5df24 slot_run + 260 
(h2_workers.c:233)
16  libsystem_pthread.dylib 0x7fffd3d1f93b _pthread_body + 180
17  libsystem_pthread.dylib 0x7fffd3d1f887 _pthread_start + 286
18  libsystem_pthread.dylib 0x7fffd3d1f08d thread_start + 13

which is a call to apr_thread_mutex_unlock(). Without having checkinto pthread 
version, the closest
similar report I could find on the net is 
https://sourceware.org/bugzilla/show_bug.cgi?id=17514

Will report when I know more.

-Stefan

> Am 27.09.2017 um 18:04 schrieb Luca Toscano :
> 
> Hi Stefan,
> 
> 2017-09-27 17:32 GMT+02:00 Stefan Eissing :
> On my h2 load tests, the server sometimes crashes in an assertion failure 
> (build in maintainer mode):
> 
> How did you make the tests? It would be good for people to attempt to 
> reproduce..
>  
> 
> macOS crash reporter:
> Thread 54 Crashed:
> 0   libsystem_kernel.dylib  0x7fffd3c34d42 __pthread_kill + 10
> 1   libsystem_pthread.dylib 0x7fffd3d22457 pthread_kill + 90
> 2   libsystem_c.dylib   0x7fffd3b9a4bb __abort + 140
> 3   libsystem_c.dylib   0x7fffd3b9a42f abort + 144
> 4   httpd   0x00010be99282 ap_log_assert + 
> 130 (log.c:1696)
> 5   mod_mpm_event.so0x00010c50b51c ap_queue_push + 188
> 6   mod_mpm_event.so0x00010c509be2 listener_thread + 
> 2226 (event.c:1749)
> 7   libsystem_pthread.dylib 0x7fffd3d1f93b _pthread_body + 180
> 8   libsystem_pthread.dylib 0x7fffd3d1f887 _pthread_start + 
> 286
> 9   libsystem_pthread.dylib 0x7fffd3d1f08d thread_start + 13
> 
> error_log:
> [Wed Sep 27 17:27:09.941040 2017] [mpm_event:notice] [pid 23404:tid 
> 140736895005632] AH00489: Apache/2.4.28 (Unix) OpenSSL/1.1.0e configured -- 
> resuming normal operations
> [Wed Sep 27 17:27:09.941055 2017] [core:notice] [pid 23404:tid 
> 140736895005632] AH00094: Command line: '/opt/apache-2.4.x/bin/httpd -d 
> /Users/sei/projects/httpd/test/mod_h2/2.4.x/gen/apache'
> [Wed Sep 27 17:29:43.207923 2017] [mpm_event:alert] [pid 30381:tid 
> 123145361334272] (35)Resource temporarily unavailable: AH03104: 
> apr_thread_create: unable to create worker thread
> [Wed Sep 27 17:29:43.208904 2017] [mpm_event:crit] [pid 30381:tid 
> 123145369382912] (9)Bad file descriptor: apr_pollset_poll failed.  Attempting 
> to shutdown process gracefully
> [Wed Sep 27 17:29:44.000147 2017] [core:notice] [pid 23404:tid 
> 140736895005632] AH00052: child pid 30381 exit signal Segmentation fault (11)
> [Wed Sep 27 17:29:44.208035 2017] [mpm_event:alert] [pid 30382:tid 
> 123145438388224] (35)Resource temporarily unavailable: AH03104: 
> apr_thread_create: unable to create worker thread
>

Re: [VOTE] Release Apache httpd 2.4.28 as GA

2017-09-28 Thread Gregg Smith 

+1 to doing a 2.4.29 next week with these issues fixed.

On 9/27/2017 9:21 PM, William A Rowe Jr wrote:

The assert() has me concerned, and Steffen's report is problematic. He has
a vote but hasn't cast it. At this moment I'm -0 and would spin a 2.4.29
next week to address these issues, unless you decide to respin before this
release, yourself.

Nothing I've changed today altered the httpd tarball significantly.
Studying Steffen's report next, along with some apparently missing glue for
brotli.

My solution is going to be radical, shove every last d*mned modules/foo
into the /I path includes list so this can't happen again during 2.4, and
hopefully not until 20 year old build logic is discarded. One less thing to
worry about or pre-review when RM's loudly announce an upcoming tag.




On Sep 25, 2017 07:13, "Jim Jagielski"  wrote:


The pre-release test tarballs for Apache httpd
version 2.4.28 can be found at the usual place:

 http://httpd.apache.org/dev/dist/

I'm calling a VOTE on releasing these as Apache httpd 2.4.28 GA.

[ ] +1: Good to go
[ ] +0: meh
[ ] -1: Danger Will Robinson. And why.

Vote will last the normal 72 hrs.

NOTE: The *-deps are only there for convenience.

Thx!