missing EOS bucket and mod_proxy_http waits forever

[himdeep pathak]

I am using apache(2.2.28) as reverse proxy

Please help me to handle following scenario..

mod_proxy_http is unable to read EOS bucket from previous output filter and it 
(ap_get_brigade()) stays on blocking state forever. 
response transfer encoding is chunk.

Missing EOS bucket could be 
1. due to buggy output filter which fails to forward EOS bucket
2. buggy back-end webserver.

how to figure out what is causing the issue and come out of infinite waiting.

Direct accessing the webserver (without apache as reverse proxy) has no issue.

Thanks
  

Re: ALPN patch comments

[Kaspar Brand]

On 27.04.2015 17:04, Stefan Eissing wrote:
 Am 25.04.2015 um 11:47 schrieb Kaspar Brand httpd-dev.2...@velox.ch:
 Only tested in terms of compiles both w/ and w/o HAVE_TLS_ALPN, so it
 certainly needs more eyes before a backport proposal could be made.
 There's also a TODO: we should have a mod_ssl configuration parameter
 in ssl_engine_kernel.c which I'm unsure to what it refers.
 
 The „TODO“ is a leftover from before SSLAlpnPreference was introduced.
 It can be removed.

I did some formatting cleanup in r1676709 and put a patch for 2.4.x
online under
https://people.apache.org/~kbrand/mod_ssl-2.4.x-alpn_2015-04-29.diff.
This should hopefully make it easier for people to test and review (it's
an amalgamation of ten revisions from trunk, see the top of the patch file).

Kaspar

Re: HEAD response's Content-Length stripped when zero

[Eric Covener]

On Wed, Apr 29, 2015 at 8:19 AM, Yann Ylavic ylavic@gmail.com wrote:
 Hence how about removing this whole block (is there any module today
 outsmarting httpd that cannot be considered as buggy?) or least
 disable it for forwarded responses, eg:

 Index: modules/http/http_filters.c
 ===
 --- modules/http/http_filters.c(revision 1676716)
 +++ modules/http/http_filters.c(working copy)
 @@ -1292,6 +1292,7 @@ AP_CORE_DECLARE_NONSTD(apr_status_t) ap_http_heade
   * The default (unset) behavior is to squelch the C-L in this case.
   */
  if (r-header_only
 + !r-proxyreq
   (clheader = apr_table_get(r-headers_out, Content-Length))
   !strcmp(clheader, 0)
   conf-http_cl_head_zero != AP_HTTP_CL_HEAD_ZERO_ENABLE) {


Maybe we could remember here if the CL was set by
ap_content_length_filter or someone else?

-- 
Eric Covener
cove...@gmail.com

Re: OpenSSL 1.02, PCRE 2.10

[Tom Browder]

On Wed, Apr 29, 2015 at 12:57 AM, Kaspar Brand httpd-dev.2...@velox.ch wrote:
 On 28.04.2015 14:04, Tom Browder wrote:
 Maybe I need to play tricks with ld.so.conf and openssl?

 Depends on whether you built OpenSSL with or without shared libraries -
 what are the contents of the /opt/openssl/lib directory?

Well, I failed to follow my original recipe and did NOT specify
'no-shared'.  I blem away openssl and used 'no-shared' and now have NO
shared libraries in /opt/openssl/lib.

 I have no system installed openssl,

 Hmm, what platform is this? Are you sure there are no libcrypto/libssl
 libraries somewhere under /usr?

I used a netinst installation of Debian 7.  But I find I do have those
libraries:

/usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.0
/usr/lib/x86_64-linux-gnu/libssl.so.1.0.0

 The most important thing, though, is making sure that the directory
 which appears in the LDFLAGS/MOD_LDFLAGS lines of the ./configure
 outputs (after the checking for user-provided OpenSSL base
 directory...) in does not include any shared library files - only
 static ones.

That is now true.

After a rebuild and re-install of openssl, I reconfigured httpd.and get this:

configure: error: Crypto was requested but no crypto library could be
enabled; specify the location of a crypto library using
--with-openssl, --with-nss, etc.

The question to me is: what exact configuration do I need?  Do I point
to a path, or library, or a configure script?  I see nothing

I've tried '--with-openssl=path' and '--with-nss=path' to no avail.

Any suggestions?

Thanks.

Best,

-Tom

Re: ALPN patch comments

[Jim Jagielski]

Thank You!!

Will review ASAP.

 On Apr 29, 2015, at 7:05 AM, Kaspar Brand httpd-dev.2...@velox.ch wrote:
 
 On 27.04.2015 17:04, Stefan Eissing wrote:
 Am 25.04.2015 um 11:47 schrieb Kaspar Brand httpd-dev.2...@velox.ch:
 Only tested in terms of compiles both w/ and w/o HAVE_TLS_ALPN, so it
 certainly needs more eyes before a backport proposal could be made.
 There's also a TODO: we should have a mod_ssl configuration parameter
 in ssl_engine_kernel.c which I'm unsure to what it refers.
 
 The „TODO“ is a leftover from before SSLAlpnPreference was introduced.
 It can be removed.
 
 I did some formatting cleanup in r1676709 and put a patch for 2.4.x
 online under
 https://people.apache.org/~kbrand/mod_ssl-2.4.x-alpn_2015-04-29.diff.
 This should hopefully make it easier for people to test and review (it's
 an amalgamation of ten revisions from trunk, see the top of the patch file).
 
 Kaspar

HEAD response's Content-Length stripped when zero

[Yann Ylavic]

Due to ap_http_header_filter():

/* This is a hack, but I can't find anyway around it.  The idea is that
 * we don't want to send out 0 Content-Lengths if it is a head request.
 * This happens when modules try to outsmart the server, and return
 * if they see a HEAD request.  Apache 1.3 handlers were supposed to
 * just return in that situation, and the core handled the HEAD.  In
 * 2.0, if a handler returns, then the core sends an EOS bucket down
 * the filter stack, and the content-length filter computes a C-L of
 * zero and that gets put in the headers, and we end up sending a
 * zero C-L to the client.  We can't just remove the C-L filter,
 * because well behaved 2.0 handlers will send their data down the stack,
 * and we will compute a real C-L for the head request. RBB
 *
 * Allow modification of this behavior through the
 * HttpContentLengthHeadZero directive.
 *
 * The default (unset) behavior is to squelch the C-L in this case.
 */
if (r-header_only
 (clheader = apr_table_get(r-headers_out, Content-Length))
 !strcmp(clheader, 0)
 conf-http_cl_head_zero != AP_HTTP_CL_HEAD_ZERO_ENABLE) {
apr_table_unset(r-headers_out, Content-Length);
}

I find this workaround a bit too much hacky because Content-Length:
0 could be the real one for the corresponding GET request, either
given by a module or forwarded from a backend.

Hence how about removing this whole block (is there any module today
outsmarting httpd that cannot be considered as buggy?) or least
disable it for forwarded responses, eg:

Index: modules/http/http_filters.c
===
--- modules/http/http_filters.c(revision 1676716)
+++ modules/http/http_filters.c(working copy)
@@ -1292,6 +1292,7 @@ AP_CORE_DECLARE_NONSTD(apr_status_t) ap_http_heade
  * The default (unset) behavior is to squelch the C-L in this case.
  */
 if (r-header_only
+ !r-proxyreq
  (clheader = apr_table_get(r-headers_out, Content-Length))
  !strcmp(clheader, 0)
  conf-http_cl_head_zero != AP_HTTP_CL_HEAD_ZERO_ENABLE) {
--
?

Re: HEAD response's Content-Length stripped when zero

[Yann Ylavic]

On Wed, Apr 29, 2015 at 2:46 PM, Eric Covener cove...@gmail.com wrote:
 On Wed, Apr 29, 2015 at 8:19 AM, Yann Ylavic ylavic@gmail.com wrote:
 Hence how about removing this whole block (is there any module today
 outsmarting httpd that cannot be considered as buggy?) or least
 disable it for forwarded responses, eg:

 Index: modules/http/http_filters.c
 ===
 --- modules/http/http_filters.c(revision 1676716)
 +++ modules/http/http_filters.c(working copy)
 @@ -1292,6 +1292,7 @@ AP_CORE_DECLARE_NONSTD(apr_status_t) ap_http_heade
   * The default (unset) behavior is to squelch the C-L in this case.
   */
  if (r-header_only
 + !r-proxyreq
   (clheader = apr_table_get(r-headers_out, Content-Length))
   !strcmp(clheader, 0)
   conf-http_cl_head_zero != AP_HTTP_CL_HEAD_ZERO_ENABLE) {


 Maybe we could remember here if the CL was set by
 ap_content_length_filter or someone else?


Yes, I first thought about this, hence checking r-clength instead,
but that would probably require to initialize it to something like
UNSET (-1), or we still could not distinguish it from a zero length
set...

That would be:

Index: modules/http/http_core.c
===
--- modules/http/http_core.c(revision 1676716)
+++ modules/http/http_core.c(working copy)
@@ -254,6 +254,9 @@ static int ap_process_http_connection(conn_rec *c)

 static int http_create_request(request_rec *r)
 {
+/* Content-Length unset by default. */
+r-clength = -1;
+
 if (!r-main  !r-prev) {
 ap_add_output_filter_handle(ap_byterange_filter_handle,
 NULL, r, r-connection);
Index: modules/http/http_filters.c
===
--- modules/http/http_filters.c(revision 1676716)
+++ modules/http/http_filters.c(working copy)
@@ -1292,6 +1292,7 @@ AP_CORE_DECLARE_NONSTD(apr_status_t) ap_http_heade
  * The default (unset) behavior is to squelch the C-L in this case.
  */
 if (r-header_only
+ (r-clength  0)
  (clheader = apr_table_get(r-headers_out, Content-Length))
  !strcmp(clheader, 0)
  conf-http_cl_head_zero != AP_HTTP_CL_HEAD_ZERO_ENABLE) {
--

Re: OpenSSL 1.02, PCRE 2.10

[Kaspar Brand]

On 28.04.2015 14:04, Tom Browder wrote:
 On Tue, Apr 28, 2015 at 6:45 AM, Eric Covener cove...@gmail.com wrote:
 about openssl 1.02 though -- what exactly do you see?
 
 I see this when attempting to start apache:
 
 /usr/local/apache2/bin/httpd: symbol lookup error:
 /usr/local/apache2/bin/httpd: undefined symbol: SSL_CONF_CTX_new
 
 Maybe I need to play tricks with ld.so.conf and openssl?

Depends on whether you built OpenSSL with or without shared libraries -
what are the contents of the /opt/openssl/lib directory?

 I have no system installed openssl,

Hmm, what platform is this? Are you sure there are no libcrypto/libssl
libraries somewhere under /usr?

 SSLDIR=/opt/openssl
 export LDFLAGS=-L${SSLDIR}/lib

This should not be needed.

 --enable-ssl   \
 --enable-ssl-staticlib-deps\
 --enable-mods-static=ssl   \
 --with-ssl=${SSLDIR}   \

--enable-mods-static=ssl is probably not really what you want - it
includes mod_ssl in the httpd binary itself, but other than that, it
doesn't have an impact on how mod_ssl is linked against libcrypto/libssl
(you might try to see what ldd /usr/local/apache2/bin/httpd says).

The most important thing, though, is making sure that the directory
which appears in the LDFLAGS/MOD_LDFLAGS lines of the ./configure
outputs (after the checking for user-provided OpenSSL base
directory...) in does not include any shared library files - only
static ones.

Kaspar

Re: HEAD response's Content-Length stripped when zero

[Yann Ylavic]

On Wed, Apr 29, 2015 at 2:46 PM, Eric Covener cove...@gmail.com wrote:
 On Wed, Apr 29, 2015 at 8:19 AM, Yann Ylavic ylavic@gmail.com wrote:
 Hence how about removing this whole block (is there any module today
 outsmarting httpd that cannot be considered as buggy?) or least
 disable it for forwarded responses, eg:

 Index: modules/http/http_filters.c
 ===
 --- modules/http/http_filters.c(revision 1676716)
 +++ modules/http/http_filters.c(working copy)
 @@ -1292,6 +1292,7 @@ AP_CORE_DECLARE_NONSTD(apr_status_t) ap_http_heade
   * The default (unset) behavior is to squelch the C-L in this case.
   */
  if (r-header_only
 + !r-proxyreq
   (clheader = apr_table_get(r-headers_out, Content-Length))
   !strcmp(clheader, 0)
   conf-http_cl_head_zero != AP_HTTP_CL_HEAD_ZERO_ENABLE) {


 Maybe we could remember here if the CL was set by
 ap_content_length_filter or someone else?

The patch below tries your proposal...

Set a note If we detect that the header_only CL is from the origin in
ap_content_length_filter(), and don't strip the Content-Length if that
note is set in ap_http_header_filter().
We could also avoid the note by setting r-clength = -1 instead in
ap_content_length_filter().

This should work both with buggy modules and the normal cases (including proxy).
I guess we wouldn't need HttpContentLengthHeadZero from r1554303
anymore (and several third-party patches that simply disable the whole
unconditionally).

Do it sound better?

Index: server/protocol.c
===
--- server/protocol.c(revision 1676716)
+++ server/protocol.c(working copy)
@@ -1533,7 +1533,7 @@ AP_CORE_DECLARE_NONSTD(apr_status_t) ap_content_le
  * We can only set a C-L in the response header if we haven't already
  * sent any buckets on to the next output filter for this request.
  */
-if (ctx-data_sent == 0  eos 
+if (ctx-data_sent == 0  eos) {
 /* don't whack the C-L if it has already been set for a HEAD
  * by something like proxy.  the brigade only has an EOS bucket
  * in this case, making r-bytes_sent zero.
@@ -1544,9 +1544,13 @@ AP_CORE_DECLARE_NONSTD(apr_status_t) ap_content_le
  * such filters update or remove the C-L header, and just use it
  * if present.
  */
-!(r-header_only  r-bytes_sent == 0 
-apr_table_get(r-headers_out, Content-Length))) {
-ap_set_content_length(r, r-bytes_sent);
+if (r-header_only  r-bytes_sent == 0 
+apr_table_get(r-headers_out, Content-Length)) {
+apr_table_setn(r-notes, origin-ho, 1);
+}
+else {
+ap_set_content_length(r, r-bytes_sent);
+}
 }

 ctx-data_sent = 1;
Index: modules/http/http_filters.c
===
--- modules/http/http_filters.c(revision 1676716)
+++ modules/http/http_filters.c(working copy)
@@ -1292,9 +1292,11 @@ AP_CORE_DECLARE_NONSTD(apr_status_t) ap_http_heade
  * The default (unset) behavior is to squelch the C-L in this case.
  */
 if (r-header_only
- (clheader = apr_table_get(r-headers_out, Content-Length))
- !strcmp(clheader, 0)
- conf-http_cl_head_zero != AP_HTTP_CL_HEAD_ZERO_ENABLE) {
+ !r-clength
+ !apr_table_get(r-notes, origin-ho)
+ (clheader = apr_table_get(r-headers_out, Content-Length))
+ !strcmp(clheader, 0)
+ conf-http_cl_head_zero != AP_HTTP_CL_HEAD_ZERO_ENABLE) {
 apr_table_unset(r-headers_out, Content-Length);
 }

--

Re: HEAD response's Content-Length stripped when zero

[Yann Ylavic]

(sorry for the patches spam / confused proposal)

On Wed, Apr 29, 2015 at 6:07 PM, Yann Ylavic ylavic@gmail.com wrote:
 We could also avoid the note by setting r-clength = -1 instead in
 ap_content_length_filter().

Another candidate could be r-sent_bodyct = 1, eg:

Index: server/protocol.c
===
--- server/protocol.c(revision 1676716)
+++ server/protocol.c(working copy)
@@ -1533,7 +1533,7 @@ AP_CORE_DECLARE_NONSTD(apr_status_t) ap_content_le
  * We can only set a C-L in the response header if we haven't already
  * sent any buckets on to the next output filter for this request.
  */
-if (ctx-data_sent == 0  eos 
+if (ctx-data_sent == 0  eos) {
 /* don't whack the C-L if it has already been set for a HEAD
  * by something like proxy.  the brigade only has an EOS bucket
  * in this case, making r-bytes_sent zero.
@@ -1544,9 +1544,16 @@ AP_CORE_DECLARE_NONSTD(apr_status_t) ap_content_le
  * such filters update or remove the C-L header, and just use it
  * if present.
  */
-!(r-header_only  r-bytes_sent == 0 
-apr_table_get(r-headers_out, Content-Length))) {
-ap_set_content_length(r, r-bytes_sent);
+if (r-header_only  r-bytes_sent == 0 
+apr_table_get(r-headers_out, Content-Length)) {
+/* Tell ap_http_header_filter() we don't want the C-L to
+ * be stripped (ie. the original no-body is sent now).
+ */
+r-sent_bodyct = 1;
+}
+else {
+ap_set_content_length(r, r-bytes_sent);
+}
 }

 ctx-data_sent = 1;
Index: modules/http/http_filters.c
===
--- modules/http/http_filters.c(revision 1676716)
+++ modules/http/http_filters.c(working copy)
@@ -1292,9 +1292,9 @@ AP_CORE_DECLARE_NONSTD(apr_status_t) ap_http_heade
  * The default (unset) behavior is to squelch the C-L in this case.
  */
 if (r-header_only
- (clheader = apr_table_get(r-headers_out, Content-Length))
- !strcmp(clheader, 0)
- conf-http_cl_head_zero != AP_HTTP_CL_HEAD_ZERO_ENABLE) {
+ !r-sent_bodyct
+ (clheader = apr_table_get(r-headers_out, Content-Length))
+ !strcmp(clheader, 0)) {
 apr_table_unset(r-headers_out, Content-Length);
 }

--

Re: Balancer manager

[William A Rowe Jr]

On Fri, Apr 24, 2015 at 8:52 AM, Jim Jagielski j...@jagunet.com wrote:

 Right now, the balancer manager allows for a member to be
 disabled/stopped, but it cannot *remove* that member...
 Seems to me that that would be good, especially since
 we could always re-use that slot.


That means you must fully implement locking against the backends table
in any use context, determine that there are no workers using that specific
backend, and then 'remove' it ('repurpose' it, really, since it's just
another
slot.)  If there are any requests in-flight for that member, this will
result
in potentially bad collisions of misinformation.

We usually ignore these mis-attributions, but I'm not sure that's feasible
given all of the lb logic involved?

Re: OpenSSL 1.02, PCRE 2.10

[Tom Browder]

On Wed, Apr 29, 2015 at 3:29 PM, William A Rowe Jr wr...@rowe-clan.net wrote:
 Just to point out the obvious, it would be a Very Bad Idea(tm) to build
 httpd against the dynamic apr-util linked statically to bits and pieces of the
...

I'm just trying to work around the current build systems to meet a
specific goal and I would appreciate anyone who can tell me EXACTLY
how to:

+ use the latest openssl
+ use the latest apr and apr-util
+ use the latest httpd

At the moment I am starting again but this time trying:

+ building and installing the latest openssl with shared libraries
+ building and installing apr and apr-util to use latest openssl
+ building and installing httpd to use all the above

Best,

-Tom

Re: OpenSSL 1.02, PCRE 2.10

[Yann Ylavic]

Hi Tom,

On Wed, Apr 29, 2015 at 11:10 PM, Tom Browder tom.brow...@gmail.com wrote:

 I'm just trying to work around the current build systems to meet a
 specific goal and I would appreciate anyone who can tell me EXACTLY
 how to:

 + use the latest openssl
 + use the latest apr and apr-util
 + use the latest httpd

Your goal is to do that dynamically now, right?


 At the moment I am starting again but this time trying:

 + building and installing the latest openssl with shared libraries
 + building and installing apr and apr-util to use latest openssl
 + building and installing httpd to use all the above

With shared libraries then, once the first point is done (say in
/opt/openssl), the simplest way is for the two others is to:
1. tar xf httpd-2.4.12.tar.bz2
2. cd httpd-2.4.12
3. tar xf extract apr-1.5.2.tar.bz2 -C srclib  mv srclib/apr-1.5.2 srclib/apr
4. tar xf extract apr-util-1.5.4.tar.bz2 -C srclib  mv
srclib/apr-util-1.5.2 srclib/apr-util
4. ./buildconf
5. ./configure --prefix=/opt/httpd --with-included-apr
--with-ssl=/opt/openssl LDFLAGS=-Wl,-rpath,/opt/openssl/lib ...
(the LDFLAGS can be omitted if you plan to somehow include
/opt/openssl/lib in LD_LIBRARY_PATH at runtime, to avoid conflct with
system's own libs)
6. make  make install

Hope I'm not off topic (you meant building statically).

Regards,
Yann.

Re: OpenSSL 1.02, PCRE 2.10

[William A Rowe Jr]

Just to point out the obvious, it would be a Very Bad Idea(tm) to build
httpd
against the dynamic apr-util linked statically to bits and pieces of the
libssl.a
and libcrypto.a, only to then attempt to build httpd with mod_ssl
compiled-in
against libssl.a/libcrypto.a again.  Unix dl logic doesn't usually play
well, this
isn't multiple-depth namespace.

It sounds a bit like the comedy routine, Doctor, it hurts when I move this
way...
so could you explain better why you are trying to accomplish what you are
trying
to accomplish?  If you rebuild and reinstall ssl in reaction to a critical
vulnerability,
the vulnerability will still be present in your httpd with mod_ssl compiled
in.  For
that reason and many others, dynamic libs are generally your friends.

The error you encountered in ./configure is that it will attempt to
compile, link
and run a short stub based on all the input parameters.  When the flags
aren't
correct to resolve the headers or libraries or run time loadable objects,
then
configure fails.  It is usually helpful to go through the config.log
results to see
just what it tried to do when it failed.




On Wed, Apr 29, 2015 at 10:56 AM, Kaspar Brand httpd-dev.2...@velox.ch
wrote:

 On 29.04.2015 15:06, Tom Browder wrote:
  On Wed, Apr 29, 2015 at 12:57 AM, Kaspar Brand httpd-dev.2...@velox.ch
 wrote:
  On 28.04.2015 14:04, Tom Browder wrote:
  I have no system installed openssl,
 
  Hmm, what platform is this? Are you sure there are no libcrypto/libssl
  libraries somewhere under /usr?
 
  I used a netinst installation of Debian 7.  But I find I do have those
  libraries:
 
  /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.0
  /usr/lib/x86_64-linux-gnu/libssl.so.1.0.0

 Ok. Debian has separate packages for the libraries and the command-line
 tool, so no system installed openssl is true as far as the openssl
 package is concerned. I'd bet that Debian is barely usable when the
 libssl1.0.0 package is missing, though.

  configure: error: Crypto was requested but no crypto library could be
  enabled; specify the location of a crypto library using
  --with-openssl, --with-nss, etc.

 This is actually APR-util configure stuff, so does not apply to mod_ssl
 itself. Unless you really have a need for mod_session_crypto, just leave
 out the --enable-crypto and --enable-session-crypto options, for the
 time being (they are not needed for mod_ssl, and are probably just
 making things more complicated than necessary right now).

  The question to me is: what exact configuration do I need?  Do I point
  to a path, or library, or a configure script?  I see nothing

 For a statically linked mod_ssl, you need --enable-ssl
 --with-ssl=/opt/openssl --enable-ssl-staticlib-deps (assuming that you
 used --prefix=/opt/openssl for OpenSSL's ./config or ./Configure).

 Kaspar

Re: OpenSSL 1.02, PCRE 2.10

[Tom Browder]

On Wed, Apr 29, 2015 at 4:51 PM, Yann Ylavic ylavic@gmail.com wrote:
 Hi Tom,

 On Wed, Apr 29, 2015 at 11:10 PM, Tom Browder tom.brow...@gmail.com wrote:

 I'm just trying to work around the current build systems to meet a
 specific goal and I would appreciate anyone who can tell me EXACTLY
 how to:

 + use the latest openssl
 + use the latest apr and apr-util
 + use the latest httpd
...

One more requirement: I want to use mod_session_crypto so I have to
'--enable-sesion-crypto' and, with Yann's options, I still get the
error:

configure: error: Crypto was requested but no crypto library could be
enabled; specify the location of a crypto library using
--with-openssl, --with-nss, etc.
configure failed for srclib/apr-util

I have not found any documentation in apr or apr-util on how to
specify the crypto library.

Thanks.

Best regards,

-Tom

Re: OpenSSL 1.02, PCRE 2.10

[Tom Browder]

On Apr 29, 2015 6:08 PM, Yann Ylavic ylavic@gmail.com wrote:

 On Thu, Apr 30, 2015 at 12:39 AM, Tom Browder tom.brow...@gmail.com
wrote:
 
  configure: error: Crypto was requested but no crypto library could be
  enabled; specify the location of a crypto library using
  --with-openssl, --with-nss, etc.
  configure failed for srclib/apr-util

 Did you try --with-openssl=/opt/openssl in addition to --with-ssl?

No, but I tried other variants--let me try that.

-Tom

Re: OpenSSL 1.02, PCRE 2.10

[Yann Ylavic]

On Thu, Apr 30, 2015 at 12:39 AM, Tom Browder tom.brow...@gmail.com wrote:

 One more requirement: I want to use mod_session_crypto so I have to
 '--enable-sesion-crypto' and, with Yann's options, I still get the
 error:

 configure: error: Crypto was requested but no crypto library could be
 enabled; specify the location of a crypto library using
 --with-openssl, --with-nss, etc.
 configure failed for srclib/apr-util

Did you try --with-openssl=/opt/openssl in addition to --with-ssl?

Re: OpenSSL 1.02, PCRE 2.10

[Yann Ylavic]

On Thu, Apr 30, 2015 at 1:57 AM, Tom Browder tom.brow...@gmail.com wrote:

 apachectl
 httpd: Syntax error on line 147 of /usr/local/apache2/conf/httpd.conf:
 Cannot load modules/mod_session_crypto.so into server:
 /usr/local/apache2/modules/mod_session_crypto.so: undefined symbol:
 ap_hook_session_encode

You probably need to also:
LoadModule session_module modules/mod_sessions.so

And use configure's --enable-session=shared if you did not already.

Regards,
Yann.

Re: OpenSSL 1.02, PCRE 2.10

[Tom Browder]

On Wed, Apr 29, 2015 at 6:30 PM, Tom Browder tom.brow...@gmail.com wrote:
 On Apr 29, 2015 6:08 PM, Yann Ylavic ylavic@gmail.com wrote:
...
 Did you try --with-openssl=/opt/openssl in addition to --with-ssl?

 No, but I tried other variants--let me try that.

Okay, now httpd is using openssl/libcrypto.

However, when I try apachectl I get:

apachectl
httpd: Syntax error on line 147 of /usr/local/apache2/conf/httpd.conf:
Cannot load modules/mod_session_crypto.so into server:
/usr/local/apache2/modules/mod_session_crypto.so: undefined symbol:
ap_hook_session_encode

It seems we are making progress, but what should be tried next.  Is
that a mod_session_crypto bug, or?

Thanks.

Best,

-Tom

Re: HEAD response's Content-Length stripped when zero

[Yann Ylavic]

On Wed, Apr 29, 2015 at 3:17 PM, Yann Ylavic ylavic@gmail.com wrote:
 On Wed, Apr 29, 2015 at 2:46 PM, Eric Covener cove...@gmail.com wrote:
 On Wed, Apr 29, 2015 at 8:19 AM, Yann Ylavic ylavic@gmail.com wrote:
 Hence how about removing this whole block (is there any module today
 outsmarting httpd that cannot be considered as buggy?) or least
 disable it for forwarded responses, eg:

 Index: modules/http/http_filters.c
 ===
 --- modules/http/http_filters.c(revision 1676716)
 +++ modules/http/http_filters.c(working copy)
 @@ -1292,6 +1292,7 @@ AP_CORE_DECLARE_NONSTD(apr_status_t) ap_http_heade
   * The default (unset) behavior is to squelch the C-L in this case.
   */
  if (r-header_only
 + !r-proxyreq
   (clheader = apr_table_get(r-headers_out, Content-Length))
   !strcmp(clheader, 0)
   conf-http_cl_head_zero != AP_HTTP_CL_HEAD_ZERO_ENABLE) {


 Maybe we could remember here if the CL was set by
 ap_content_length_filter or someone else?


 Yes, I first thought about this, hence checking r-clength instead,

Hmm, I have read ap_content_length_filter() instead of
ap_set_content_length(), please forget about my previous answer(s)...
I need to think more about this.

Re: HEAD response's Content-Length stripped when zero

[Yann Ylavic]

On Wed, Apr 29, 2015 at 3:39 PM, Yann Ylavic ylavic@gmail.com wrote:
 On Wed, Apr 29, 2015 at 3:17 PM, Yann Ylavic ylavic@gmail.com wrote:
 On Wed, Apr 29, 2015 at 2:46 PM, Eric Covener cove...@gmail.com wrote:
 On Wed, Apr 29, 2015 at 8:19 AM, Yann Ylavic ylavic@gmail.com wrote:
 Hence how about removing this whole block (is there any module today
 outsmarting httpd that cannot be considered as buggy?) or least
 disable it for forwarded responses, eg:

 Index: modules/http/http_filters.c
 ===
 --- modules/http/http_filters.c(revision 1676716)
 +++ modules/http/http_filters.c(working copy)
 @@ -1292,6 +1292,7 @@ AP_CORE_DECLARE_NONSTD(apr_status_t) ap_http_heade
   * The default (unset) behavior is to squelch the C-L in this case.
   */
  if (r-header_only
 + !r-proxyreq
   (clheader = apr_table_get(r-headers_out, Content-Length))
   !strcmp(clheader, 0)
   conf-http_cl_head_zero != AP_HTTP_CL_HEAD_ZERO_ENABLE) {


 Maybe we could remember here if the CL was set by
 ap_content_length_filter or someone else?


 Yes, I first thought about this, hence checking r-clength instead,

 Hmm, I have read ap_content_length_filter() instead of
 ap_set_content_length(), please forget about my previous answer(s)...
 I need to think more about this.

Actually I found http://svn.apache.org/r1554303 which make it configurable.
Maybe we can backport it instead (why a MAJOR bump?? I'd like to see
it in 2.2.x too...).
I'd argue that the default should be HttpContentLengthHeadZero on
though (at least in trunk), still users of buggy/legacy modules could
turn it off.

Re: HEAD response's Content-Length stripped when zero

[Yann Ylavic]

On Wed, Apr 29, 2015 at 3:17 PM, Yann Ylavic ylavic@gmail.com wrote:
 Index: modules/http/http_filters.c
 ===
 --- modules/http/http_filters.c(revision 1676716)
 +++ modules/http/http_filters.c(working copy)
 @@ -1292,6 +1292,7 @@ AP_CORE_DECLARE_NONSTD(apr_status_t) ap_http_heade
   * The default (unset) behavior is to squelch the C-L in this case.
   */
  if (r-header_only
 + (r-clength  0)

We probably still require:
+ !r-proxyreq
since mod_proxy does not call ap_set_content_length().

Prefetch before connecting/reusing backend and leverage ping/100-continue (was: svn commit: r1676855 - /httpd/httpd/branches/2.4.x/STATUS)

[Yann Ylavic]

On Wed, Apr 29, 2015 at 10:48 PM,  yla...@apache.org wrote:

 +   * mod_proxy_http: Don't establish or reuse a backend connection before 
 pre-
 + fetching the request body, so to minimize the delay between it is 
 supposed
 + to be alive and the first bytes sent: this is a best effort to prevent 
 the
 + backend from closing because of idle or keepalive timeout in the 
 meantime.
 + Also, handle a new proxy-flushall environment variable which allows to
 + flush any forwarded body data immediately. PR 56541+37920.
 + trunk patch: http://svn.apache.org/r1656259
 +  http://svn.apache.org/r1656359 (CHANGES entry)
 + 2.4.x patch: trunk works (modulo CHANGES, docs/log-message-tags)
 + +1: ylavic
 + -0: jim:  This seems to be a hit to normal performance, to handle an
 +   error and/or non-normal condition. The pre-fetch is
 +   expensive, and is always done, even before we know that
 +   the backend is available to rec' it. I understand the
 +   error described, but is the fix actually worth it (plus
 +   it seems to allow for a DDoS vector).
 + ylavic: It seems to me that the problem is real since we reuse the
 + connection before prefetching 16K (either controlled by the
 + client, or by an input filter), we currently always prefetch
 + these bytes already. Regarding performance I don't see any
 + difference (more cycles) compared with the current code.
 + However I think I failed to rebuild the header_brigade when
 + the proxy loop is retried (ping), so I need to rework this.
 + Do you think we'd better remove the prefetch, or maybe just
 + make it nonblocking (by default)?
 +jim: Non-blocking seems the best way to handle...

The prefetch, AIUI, was (before r1656259) and is still is here to help
determine C-L vs T-E (which may not be accepted on the backend side)
or spool (which is costly) for the request, at least with bodies 
64K.
If we were to use non-blocking for it, I guess we would lose this
ability whenever the client's header and body are not sent/received at
the same time...
And I don't think this is necessary, at no cost (AFAICT).

Can you please elaborate on the performance hit and DDoS?
Regarding performance, the new code is just a reordering of the steps
compared to the old one.
Regarding DDoS, you seem to refer to the fact that we now prefetch
(and may spool) the request body before we know whether or not the
backend is available, but is that exploitable without also
controlling/downing the backend?
And in that case, was the previous code much better wrt
ap_discard_request_body() (which is finally called after the error
response)?
I'm not saying this is not a concern, it is surely, and I think the
new changes proposed below can address this issue too.

So I reworked the idea to prefetch before connecting/reusing the
backend/connection (in blocking mode still).
To address the issue regarding the retry (ping) loop and the
{header,input/prefetch}_brigade being consumed on the first try:
- the header brigade can be rebuilt when ping request is retried (by
calling ap_proxy_create_hdrbrgd() again, as before r1656259),
- this is not the case for the input/prefetch brigade (which was also
buggy before r1656259), since we don't want to preventively set it
aside (that was discussed in a previous thread IIRC).

Hence we need to the ability to send the header first, check for
success and continue with the body, or retry with another connection
(and keep the body for it).
To acheive that, we could split stream_reqbody_{cl,chunked}() in two,
the existing ones for the body only, and the new
send_reqhdr_{cl,chunked}() for the header (note that
spool_reqbody_cl() is called during prefetch, so we only need to make
it provide two separate header and input brigades which we can
ap_proxy_pass_brigade() at any time).
Then, once the body is prefetched, the backend is reused/connected, we
could first send the header and either stream/pass the body immediatly
(no ping configured), or go with ap_proxy_http_process_response()
which will stream/pass it once the 100-continue is received, or fail
(and be retried once).

Note that this also allows for end-to-end 100-continue, since we could
forward the 100-continue to the client before streaming its body (no
need to prefetch at all if the client expects 100-continue!).

I agree that this could be done either with prefetch done before or
after connecting/reusing the backend, but since before is (much) less
racy wrt backend and the above can possibly avoid the cons, I'd prefer
to keep it.

I'm almost done with these changes and will propose a patch soon,
thoughts/showstoppers?

Re: OpenSSL 1.02, PCRE 2.10

[Tom Browder]

On Wed, Apr 29, 2015 at 7:09 PM, Yann Ylavic ylavic@gmail.com wrote:
 On Thu, Apr 30, 2015 at 1:57 AM, Tom Browder tom.brow...@gmail.com wrote:

 apachectl
 httpd: Syntax error on line 147 of /usr/local/apache2/conf/httpd.conf:
 Cannot load modules/mod_session_crypto.so into server:
 /usr/local/apache2/modules/mod_session_crypto.so: undefined symbol:
 ap_hook_session_encode

 You probably need to also:
 LoadModule session_module modules/mod_sessions.so

First I tried

  --enable-session

which is supposed to have it enabled in the httpd.conf file but it
doesn't (a bug).

But the '--enable-load-all-modules' does work!

I also found that the https.conf file wasn't changing during my config
changes (I believe that warrants a better note and perhaps another
option such as '--force' which recreates all when experimenting as I
have been doing).  I have to remove them before installation.

But, after I manually uncommented the session module in the httpd.conf
file all seems to work fine and I am able to start my shiny new httpd.

Thanks so much Yann for bearing with me through this useful exercise!

Warmest regards,

-Tom

Proposal/RFC: informed load balancing

[Jim Riggs]

[ Long message and proposal follows. Bear with me. There are a lot of words, 
but that is because we need a lot of help/input! ;-) ]

So, this has come up in the past several times, and we discussed it again this 
year at ApacheCon: How do we get the load balancer to make smarter, more 
informed decisions about where to send traffic?

The different LB methods provide some different attempts at balancing traffic, 
but ultimately none of them is smart about its decision. Other than a member 
being in error state, the balancer makes its decision solely based on 
configuration (LB set, factor, etc.) and its own knowledge of the member (e.g. 
requests, bytes). What we have often discussed is a way to get some type of 
health/load/capacity information from the backend to make informed balancing 
decisions.

One method is to use health checks (a la haproxy, AWS ELBs, etc.) that request 
one or more URLs and the response code/time indicates whether or not the 
service is up and available, allowing more proactive decisions. While this is 
better than our current state of reactively marking members in error state 
based on failed requests, it still provides a limited view of the health/state 
of the backend.

We have also discussed implementing a way for backends to communicate a magical 
load number to the front end to take into account as it balances traffic. 
This would give a much better view into the backend's state, but requires some 
way to come up with this calculation that each backend 
system/server/service/app must provide. This then has to be implemented in all 
the various backends (e.g. httpd, tomcat, php-fpm, unicorn, mongrel, etc., 
etc.), probably a hard sell to all of those projects. And, the front end would 
have limited control over what that number means or how to use it.

During JimJag's balancer talk at ApacheCon this year, he brought up this issue 
of better, more informed decision making again. I put some thought into it 
that night and came up with some ideas. Jim, Covener, Trawick, Ruggeri, and I 
then spent some time over the next couple of days talking it through and 
fleshing out some of the details.

Based on all of that, below is what I am proposing. I have some initial code 
that I am working on to implement the different pieces of this, and I will put 
them up in bugz or somewhere when they're a little less rudimentary.

--

Our hope is to create a general standard that can be used by various projects, 
products, proxies, servers, etc., to have a more consistent way for a load 
balancer to request and receive useful internal state information from its 
backend nodes. This information can then be used by the *frontend* 
software/admin (this is the main change from what we have discussed before) to 
calculate a load factor appropriate for each backend node.

This communication uses a new, standard HTTP header, X-Backend-Info, that 
takes this form in RFC2616 BNF:

backend-info  = version = numeric-entry
[
  *LWS , *LWS
  #( numeric-entry | string-entry )
]

numeric-entry = numeric-field = ( float |  float  )
; that is, numbers may optionally be enclosed in
; quotation marks

float = 1*DIGIT [ . 1*DIGIT ]

numeric-field = workers-max
; maximum number of workers the backend supports
  | workers-used
; current number of used/busy workers
  | workers-allocated
; current number of allocated/ready workers
  | workers-free
; current number of workers available for use
; (generally the difference between workers-max and
; workers-used, though some implementations may have
; a different notion)
  | uptime
; number of seconds the backend has been running
  | requests
; number of requests the backend has processed
  | memory-max
; total amount of memory available in bytes
  | memory-used
; amount of used memory in bytes
  | memory-allocated
; amount of allocated/committed memory in bytes
  | memory-free
; amount of memory available for use (generally
; the difference between memory-max and memory-used,
; though some implementations may have a different
; notion)
  | load-current
; the (subjective) current load for the backend
  | load-5
; the (subjective) 5-minute load for the backend
  | load-15
; the (subjective) 15-minute load for the backend

string-entry  = string-field = ( token | 

Re: OpenSSL 1.02, PCRE 2.10

[Kaspar Brand]

On 29.04.2015 15:06, Tom Browder wrote:
 On Wed, Apr 29, 2015 at 12:57 AM, Kaspar Brand httpd-dev.2...@velox.ch 
 wrote:
 On 28.04.2015 14:04, Tom Browder wrote:
 I have no system installed openssl,

 Hmm, what platform is this? Are you sure there are no libcrypto/libssl
 libraries somewhere under /usr?
 
 I used a netinst installation of Debian 7.  But I find I do have those
 libraries:
 
 /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.0
 /usr/lib/x86_64-linux-gnu/libssl.so.1.0.0

Ok. Debian has separate packages for the libraries and the command-line
tool, so no system installed openssl is true as far as the openssl
package is concerned. I'd bet that Debian is barely usable when the
libssl1.0.0 package is missing, though.

 configure: error: Crypto was requested but no crypto library could be
 enabled; specify the location of a crypto library using
 --with-openssl, --with-nss, etc.

This is actually APR-util configure stuff, so does not apply to mod_ssl
itself. Unless you really have a need for mod_session_crypto, just leave
out the --enable-crypto and --enable-session-crypto options, for the
time being (they are not needed for mod_ssl, and are probably just
making things more complicated than necessary right now).

 The question to me is: what exact configuration do I need?  Do I point
 to a path, or library, or a configure script?  I see nothing

For a statically linked mod_ssl, you need --enable-ssl
--with-ssl=/opt/openssl --enable-ssl-staticlib-deps (assuming that you
used --prefix=/opt/openssl for OpenSSL's ./config or ./Configure).

Kaspar