C-L filter buffering snafu

[Joe Orton]

https://bz.apache.org/bugzilla/show_bug.cgi?id=61222

I may be missing something here, ap_content_length_filter looks broken. 
Currently it implements an unlimited size buffer, by trying to morph 
every indeterminate length bucket into the heap.  It has the standard 
"read till it blocks then flush" logic, but this isn't a defence against 
RAM consumption!

If the (e.g) CGI script is fast enough that read()s never block it just 
keeps on sucking up HEAP buckets, as the simple repro case in the bug 
shows.

1. am I being stupid here?

2. otherwise, is there a good defence of why that filter should buffer 
to try to compute a C-L - exactly how much it should buffer & why?!

Regards, Joe

Interest in a uwsgi mod_proxy module?

[Jim Jagielski]

I am thinking about putting together a mod_proxy_uwsgi to
complement our current roster of (sub)modules... I don't
want to spend the cycles unless there is some interest in
having such a beast in our core distro.

Is there?

Re: svn commit: r1807777 - /httpd/httpd/trunk/modules/md/Makefile.in

[Stefan Eissing]

Thanks!

> Am 11.09.2017 um 12:22 schrieb Luca Toscano :
> 
> 
> 
> 2017-09-11 12:20 GMT+02:00 Yann Ylavic :
> On Mon, Sep 11, 2017 at 12:14 PM, Yann Ylavic  wrote:
> >
> > Index: modules/md/Makefile.in
> > ===
> > --- modules/md/Makefile.in  (revision 1807999)
> > +++ modules/md/Makefile.in  (working copy)
> > @@ -41,7 +41,7 @@ a2md_OBJECTS = \
> > md_cmd_reg.c \
> > md_cmd_store.c
> >
> > -a2md: $(a2md_OBJECTS) $(COMMON_SHOBJECTS) md.h
> > +a2md: $(a2md_OBJECTS) $(COMMON_OBJECTS) $(COMMON_SHOBJECTS) md.h
> > $(LINK) $(a2md_CFLAGS) $(a2md_LTFLAGS) $(a2md_OBJECTS)
> > $(COMMON_OBJECTS) $(A2MD_LDADD) $(AP_LIBS)
> >
> >  # top be installed in bin dir
> > _
> >
> > Looks good to commit?
> 
> Committed in r1808005 for a better look (gmail mangled the above).
> 
> Fixed the warning for me! Thanks!
> 
> Luca 
> 

Re: svn commit: r1807777 - /httpd/httpd/trunk/modules/md/Makefile.in

[Luca Toscano]

2017-09-11 12:20 GMT+02:00 Yann Ylavic :

> On Mon, Sep 11, 2017 at 12:14 PM, Yann Ylavic 
> wrote:
> >
> > Index: modules/md/Makefile.in
> > ===
> > --- modules/md/Makefile.in  (revision 1807999)
> > +++ modules/md/Makefile.in  (working copy)
> > @@ -41,7 +41,7 @@ a2md_OBJECTS = \
> > md_cmd_reg.c \
> > md_cmd_store.c
> >
> > -a2md: $(a2md_OBJECTS) $(COMMON_SHOBJECTS) md.h
> > +a2md: $(a2md_OBJECTS) $(COMMON_OBJECTS) $(COMMON_SHOBJECTS) md.h
> > $(LINK) $(a2md_CFLAGS) $(a2md_LTFLAGS) $(a2md_OBJECTS)
> > $(COMMON_OBJECTS) $(A2MD_LDADD) $(AP_LIBS)
> >
> >  # top be installed in bin dir
> > _
> >
> > Looks good to commit?
>
> Committed in r1808005 for a better look (gmail mangled the above).
>

Fixed the warning for me! Thanks!

Luca

Re: svn commit: r1807777 - /httpd/httpd/trunk/modules/md/Makefile.in

[Yann Ylavic]

On Mon, Sep 11, 2017 at 12:14 PM, Yann Ylavic  wrote:
>
> Index: modules/md/Makefile.in
> ===
> --- modules/md/Makefile.in  (revision 1807999)
> +++ modules/md/Makefile.in  (working copy)
> @@ -41,7 +41,7 @@ a2md_OBJECTS = \
> md_cmd_reg.c \
> md_cmd_store.c
>
> -a2md: $(a2md_OBJECTS) $(COMMON_SHOBJECTS) md.h
> +a2md: $(a2md_OBJECTS) $(COMMON_OBJECTS) $(COMMON_SHOBJECTS) md.h
> $(LINK) $(a2md_CFLAGS) $(a2md_LTFLAGS) $(a2md_OBJECTS)
> $(COMMON_OBJECTS) $(A2MD_LDADD) $(AP_LIBS)
>
>  # top be installed in bin dir
> _
>
> Looks good to commit?

Committed in r1808005 for a better look (gmail mangled the above).

Re: svn commit: r1807777 - /httpd/httpd/trunk/modules/md/Makefile.in

[Yann Ylavic]

On Mon, Sep 11, 2017 at 10:23 AM, Stefan Eissing
 wrote:
>
>> Am 08.09.2017 um 17:14 schrieb Yann Ylavic :
>>
>> Hi Stefan,
>>
>> On Fri, Sep 8, 2017 at 5:06 PM,   wrote:
>>> Author: icing
>>> Date: Fri Sep  8 15:06:44 2017
>>> New Revision: 180
>>>
>>> URL: http://svn.apache.org/viewvc?rev=180=rev
>>> Log:
>>> On the trunk:
>>>
>>> mod_md: added necessary CPPFLAGS for a2md build.
>>
>> Thanks, it fixed some APR dependencies for me.
>>
>> Now I'm here:
>>
>> || *** Warning: Linking the shared library mod_md.la against the non-libtool
>> || *** objects  md_acme.o md_acme_acct.o md_acme_authz.o
>> md_acme_drive.o md_core.o md_curl.o md_crypt.o md_http.o md_json.o
>> md_jws.o md_log.o md_reg.o md_store.o md_store_fs.o md_util.o is not
>> portable!
>> || /home/yle/bin/ld: error: md_json.o: requires dynamic R_X86_64_PC32
>> reloc against 'md_json_destroy' which may overflow at runtime;
>> recompile with -fPIC
>> || /home/yle/bin/ld: error: md_reg.o: requires dynamic R_X86_64_PC32
>> reloc against 'md_reg_find_overlap' which may overflow at runtime;
>> recompile with -fPIC
>> || /home/yle/bin/ld: error: md_util.o: requires dynamic R_X86_64_PC32
>> reloc against 'md_array_str_index' which may overflow at runtime;
>> recompile with -fPIC
>> || collect2: error: ld returned 1 exit status
>>
>> Any idea?
>
> Really did a buildconf && configure && make clean? This worked
> for others...

Yes, but then make -j8...
Looks like a dependencies race, and indeed with the below it now works with -j.

Index: modules/md/Makefile.in
===
--- modules/md/Makefile.in  (revision 1807999)
+++ modules/md/Makefile.in  (working copy)
@@ -41,7 +41,7 @@ a2md_OBJECTS = \
md_cmd_reg.c \
md_cmd_store.c

-a2md: $(a2md_OBJECTS) $(COMMON_SHOBJECTS) md.h
+a2md: $(a2md_OBJECTS) $(COMMON_OBJECTS) $(COMMON_SHOBJECTS) md.h
$(LINK) $(a2md_CFLAGS) $(a2md_LTFLAGS) $(a2md_OBJECTS)
$(COMMON_OBJECTS) $(A2MD_LDADD) $(AP_LIBS)

 # top be installed in bin dir
_

Looks good to commit?

Re: svn commit: r1807777 - /httpd/httpd/trunk/modules/md/Makefile.in

[Luca Toscano]

2017-09-11 10:23 GMT+02:00 Stefan Eissing :

>
> > Am 08.09.2017 um 17:14 schrieb Yann Ylavic :
> >
> > Hi Stefan,
> >
> > On Fri, Sep 8, 2017 at 5:06 PM,   wrote:
> >> Author: icing
> >> Date: Fri Sep  8 15:06:44 2017
> >> New Revision: 180
> >>
> >> URL: http://svn.apache.org/viewvc?rev=180=rev
> >> Log:
> >> On the trunk:
> >>
> >> mod_md: added necessary CPPFLAGS for a2md build.
> >
> > Thanks, it fixed some APR dependencies for me.
> >
> > Now I'm here:
> >
> > || *** Warning: Linking the shared library mod_md.la against the
> non-libtool
> > || *** objects  md_acme.o md_acme_acct.o md_acme_authz.o
> > md_acme_drive.o md_core.o md_curl.o md_crypt.o md_http.o md_json.o
> > md_jws.o md_log.o md_reg.o md_store.o md_store_fs.o md_util.o is not
> > portable!
> > || /home/yle/bin/ld: error: md_json.o: requires dynamic R_X86_64_PC32
> > reloc against 'md_json_destroy' which may overflow at runtime;
> > recompile with -fPIC
> > || /home/yle/bin/ld: error: md_reg.o: requires dynamic R_X86_64_PC32
> > reloc against 'md_reg_find_overlap' which may overflow at runtime;
> > recompile with -fPIC
> > || /home/yle/bin/ld: error: md_util.o: requires dynamic R_X86_64_PC32
> > reloc against 'md_array_str_index' which may overflow at runtime;
> > recompile with -fPIC
> > || collect2: error: ld returned 1 exit status
> >
> > Any idea?
>
> Really did a buildconf && configure && make clean? This worked
> for others...


Just tried and this is what I get:

./buildconf --with-apr=srclib/apr --with-apr-util=srclib/apr-uti
./configure --with-included-apr --enable-mods-shared=most
make clean && make

Debian Jessie (Vagrant image)

/home/vagrant/httpd-trunk/srclib/apr/libtool --silent --mode=link gcc
-I/usr/include/x86_64-linux-gnu -g -O2 -pthread-lssl -lcrypto  -luuid
-lrt -lcrypt  -lpthread -ldl -ljansson -lcurl   -o mod_md.la -rpath
/usr/local/apache2/modules -module -avoid-version  mod_md.lo
mod_md_config.lo mod_md_os.lo  md_acme.o md_acme_acct.o md_acme_authz.o
md_acme_drive.o md_core.o md_curl.o md_crypt.o md_http.o md_json.o md_jws.o
md_log.o md_reg.o md_store.o md_store_fs.o md_util.o

*** Warning: Linking the shared library mod_md.la against the non-libtool
*** objects  md_acme.o md_acme_acct.o md_acme_authz.o md_acme_drive.o
md_core.o md_curl.o md_crypt.o md_http.o md_json.o md_jws.o md_log.o
md_reg.o md_store.o md_store_fs.o md_util.o is not portable!

Luca

Re: svn commit: r1807777 - /httpd/httpd/trunk/modules/md/Makefile.in

[Stefan Eissing]

> Am 08.09.2017 um 17:14 schrieb Yann Ylavic :
> 
> Hi Stefan,
> 
> On Fri, Sep 8, 2017 at 5:06 PM,   wrote:
>> Author: icing
>> Date: Fri Sep  8 15:06:44 2017
>> New Revision: 180
>> 
>> URL: http://svn.apache.org/viewvc?rev=180=rev
>> Log:
>> On the trunk:
>> 
>> mod_md: added necessary CPPFLAGS for a2md build.
> 
> Thanks, it fixed some APR dependencies for me.
> 
> Now I'm here:
> 
> || *** Warning: Linking the shared library mod_md.la against the non-libtool
> || *** objects  md_acme.o md_acme_acct.o md_acme_authz.o
> md_acme_drive.o md_core.o md_curl.o md_crypt.o md_http.o md_json.o
> md_jws.o md_log.o md_reg.o md_store.o md_store_fs.o md_util.o is not
> portable!
> || /home/yle/bin/ld: error: md_json.o: requires dynamic R_X86_64_PC32
> reloc against 'md_json_destroy' which may overflow at runtime;
> recompile with -fPIC
> || /home/yle/bin/ld: error: md_reg.o: requires dynamic R_X86_64_PC32
> reloc against 'md_reg_find_overlap' which may overflow at runtime;
> recompile with -fPIC
> || /home/yle/bin/ld: error: md_util.o: requires dynamic R_X86_64_PC32
> reloc against 'md_array_str_index' which may overflow at runtime;
> recompile with -fPIC
> || collect2: error: ld returned 1 exit status
> 
> Any idea?

Really did a buildconf && configure && make clean? This worked
for others...

Re: svn commit: r1807777 - /httpd/httpd/trunk/modules/md/Makefile.in

[Stefan Eissing]

Yann, thanks for testing. I had the build working on MacOS, but gcc on my
Ubuntu was not happy. The problem is something that plagued me already in
mod_http2: having the same source/objects shared in two link targets.

Previously, I build libmd.la and linked that to mod_md and a2md (the cmd 
line util). libmd.la's objects were built with -prefer-non-pic by our
build system and linking that into mod_md.so was refused by Linux linker.

I did not manage to build a libmd.* that then gets statically linked into
mod_md and a2md and uses pic.

So, I added the objects themselves to the targets of mod_md and a2md. That
made it working on Linux (Ubuntu) and MacOS for me.

Why linking statically? I did not want to have to install a libmd.so in
the system and manage that. I just have some common code between a
module and a cmd line tool.

I'd really appreciate if someone with a deeper understanding of our
ancient build system could come up with a solution to this.

> Am 08.09.2017 um 17:43 schrieb William A Rowe Jr :
> 
> On Fri, Sep 8, 2017 at 10:14 AM, Yann Ylavic  wrote:
>> Hi Stefan,
>> 
>> On Fri, Sep 8, 2017 at 5:06 PM,   wrote:
>>> Author: icing
>>> Date: Fri Sep  8 15:06:44 2017
>>> New Revision: 180
>>> 
>>> URL: http://svn.apache.org/viewvc?rev=180=rev
>>> Log:
>>> On the trunk:
>>> 
>>> mod_md: added necessary CPPFLAGS for a2md build.
>> 
>> Thanks, it fixed some APR dependencies for me.
>> 
>> Now I'm here:
>> 
>> || *** Warning: Linking the shared library mod_md.la against the non-libtool
>> || *** objects  md_acme.o md_acme_acct.o md_acme_authz.o
>> md_acme_drive.o md_core.o md_curl.o md_crypt.o md_http.o md_json.o
>> md_jws.o md_log.o md_reg.o md_store.o md_store_fs.o md_util.o is not
>> portable!
>> || /home/yle/bin/ld: error: md_json.o: requires dynamic R_X86_64_PC32
>> reloc against 'md_json_destroy' which may overflow at runtime;
>> recompile with -fPIC
>> || /home/yle/bin/ld: error: md_reg.o: requires dynamic R_X86_64_PC32
>> reloc against 'md_reg_find_overlap' which may overflow at runtime;
>> recompile with -fPIC
>> || /home/yle/bin/ld: error: md_util.o: requires dynamic R_X86_64_PC32
>> reloc against 'md_array_str_index' which may overflow at runtime;
>> recompile with -fPIC
>> || collect2: error: ld returned 1 exit status
>> 
>> Any idea?

I assume you had done a make clean? Those objects should have been build using 
pic now.

buildbot success in on httpd-trunk

[buildbot]

The Buildbot has detected a restored build on builder httpd-trunk while 
building . Full details are available at:
https://ci.apache.org/builders/httpd-trunk/builds/1025

Buildbot URL: https://ci.apache.org/

Buildslave for this Build: bb_slave6_ubuntu

Build Reason: The AnyBranchScheduler scheduler named 'httpd-trunk-on-commit' 
triggered this build
Build Source Stamp: [branch httpd/httpd/trunk] 1807994
Blamelist: rpluem

Build succeeded!

Sincerely,
 -The Buildbot

buildbot failure in on httpd-trunk

[buildbot]

The Buildbot has detected a new failure on builder httpd-trunk while building . 
Full details are available at:
https://ci.apache.org/builders/httpd-trunk/builds/1024

Buildbot URL: https://ci.apache.org/

Buildslave for this Build: bb_slave6_ubuntu

Build Reason: The AnyBranchScheduler scheduler named 'httpd-trunk-on-commit' 
triggered this build
Build Source Stamp: [branch httpd/httpd/trunk] 1807993
Blamelist: dferradal,lgentis,rpluem

BUILD FAILED: failed compile

Sincerely,
 -The Buildbot