AW: mod_jk interference with ErrorDocument/Alias on HEAD request
Hello Stefan,
the debug output of mod_jk shows at least which route the request is going:
[info] jk_handler::mod_jk.c (2968): No body with status=401 for
worker=ajp13_worker
So it looks like that the code
https://github.com/apache/tomcat-connectors/blob/main/native/apache-2.0/mod_jk.c#L2954#L2973
Returns 401 to the caller.
Apache sets the flag header_only when receiving a HEAD-Request. This flag can
also be seen in the mod_jk sources.
The "Excess" error is produces by curl:
https://fossies.org/linux/curl/lib/transfer.c
Dunno if this info helps much.
Greetings, Thomas
-Ursprüngliche Nachricht-
Von: Stefan Mayr
Gesendet: Dienstag, 15. Februar 2022 14:26
An: users@tomcat.apache.org
Betreff: Re: mod_jk interference with ErrorDocument/Alias on HEAD request
Hello Thomas,
Am 15.02.2022 um 11:38 schrieb Thomas Hoffmann (Speed4Trade GmbH):
> Hello Stefan,
>
> by spec / RFC, a HEAD request is not allowed to return any body.
>
> Greetings,
> Thomas
This is true and that is why i'm writing to this list. In the described case
mod_jk returns a response body although it should not (at least i think mod_jk
is somehow responsible for that)
> -Ursprüngliche Nachricht-
> Von: Stefan Mayr
> Gesendet: Montag, 14. Februar 2022 23:07
> An: users@tomcat.apache.org
> Betreff: Re: mod_jk interference with ErrorDocument/Alias on HEAD
> request
>
> Hello again,
>
> a self-compiled mod_jk 1.2.48 shows the same issue.
>
> Am 13.02.2022 um 18:37 schrieb Stefan Mayr:
>> Hi,
>>
>> looking at the source code
>> https://github.com/apache/tomcat-connectors/blob/main/native/apache-2.
>> 0/mod_jk.c#L2954#L2973
>> I did some more testing:
>>
>> Variant 1: JkMount /demo/* ajp13_worker;use_server_errors=401
>> Variant 2: JkMount /demo/* ajp13_worker
>>
>> ignoring what variant 2 changes for regular request: reading the
>> source comment my understanding is, that for both variants a HEAD
>> request (by definition must have an empty response body) should let
>> Apache httpd handle the error code.
>>
>> But the return code for jk_handler looks different:
>>
>> Variant 1: s.http_response_status
>> Variant 2: r->status
>
> Although this looks different on the first glance it seems to be the same.
>
>> The response only seems correct for variant 1 - which is configured
>> to let Apache httpd handle all responses for status codes >= 401. For
>> variant 2 mod_jk seems to handle the response itself - contrary to
>> what the comment explains.
>
> This leads to the next assumption, that whenever there is a special handling
> for use_server_errors there should be something similar for the case with an
> empty/non-existing response body.
>
> There is
> https://github.com/apache/tomcat-connectors/blob/main/native/common/jk
> _ajp_common.c#L1991#L1993 with no corresponding (pseudo) code like
>
> if (!r->something_like_bodyct && r->http_response_status >=
> JK_HTTP_BAD_REQUEST){
> r->response_blocked = JK_TRUE;
> }
>
> Adding code like this (sorry, i could not find out how to determine if there
> is a response body) fixes the issue with the wrong response body for a HEAD
> request. But we miss the WWW-Authenticate header now.
>
> Digging further we find
> https://github.com/apache/tomcat-connectors/blob/main/native/apache-2.
> 0/mod_jk.c#L331#L353 which has a special treatment for 401 HTTP
> Unauthorized. But again, only for use_server_errors.
>
> This should be fixable by extending the condition like this
>
> if ((s->extension.use_server_error_pages &&
> status >= s->extension.use_server_error_pages) ||
> (!r->sent_bodyct && r->status >= HTTP_BAD_REQUEST)) {
>
>
> But the WWW-Authenticate header is still missing. So i'm wrong, again.
> Although it feels like i'm close.
>
>> Am 12.02.2022 um 14:24 schrieb Stefan Mayr:
>>> Hello Tomcat users,
>>>
>>> this week we were debugging a strange connection issue which I
>>> tracked down to an interference between Apache httpd and mod_jk.
>>>
>>> For the full picture, the infrastructure setup contains
>>>
>>> 1. a Loadbalancer providing HTTPS, HTTP/1.1 and HTTP/2 for Clients.
>>> 2. an Apache httpd 2.4 webserver (HTTP only) with mod_jk 3. a Tomcat
>>> mit AJP-Connector
>>>
>>> We have an application doing many different HEAD requests against an
>>> application running in the Tomcat server. The requests contain an
>>> Authorization header for Basic authentication. Expected response is
>>> a HTTP 200 OK or HTTP 401 if this particular user is not allowed to
>>> access that ressource. Because this is a HEAD request there must not
>>> be a response body according to RFC 2616.
>>>
>>> If there is a response body in the response to the HEAD request our
>>> loadbalancer does strange things: aborts the connection if the
>>> clients uses HTTP/2, SSL errors if using HTTP/1.1. But this is an
>>> issue on its own which we might have to solve with the vendor.
>>>
>>> Now comes the
AW: mod_jk interference with ErrorDocument/Alias on HEAD request
Hello Stefan,
Now I got it. Thanks for the clarification :)
-Ursprüngliche Nachricht-
Von: Stefan Mayr
Gesendet: Dienstag, 15. Februar 2022 14:26
An: users@tomcat.apache.org
Betreff: Re: mod_jk interference with ErrorDocument/Alias on HEAD request
Hello Thomas,
Am 15.02.2022 um 11:38 schrieb Thomas Hoffmann (Speed4Trade GmbH):
> Hello Stefan,
>
> by spec / RFC, a HEAD request is not allowed to return any body.
>
> Greetings,
> Thomas
This is true and that is why i'm writing to this list. In the described case
mod_jk returns a response body although it should not (at least i think mod_jk
is somehow responsible for that)
> -Ursprüngliche Nachricht-
> Von: Stefan Mayr
> Gesendet: Montag, 14. Februar 2022 23:07
> An: users@tomcat.apache.org
> Betreff: Re: mod_jk interference with ErrorDocument/Alias on HEAD
> request
>
> Hello again,
>
> a self-compiled mod_jk 1.2.48 shows the same issue.
>
> Am 13.02.2022 um 18:37 schrieb Stefan Mayr:
>> Hi,
>>
>> looking at the source code
>> https://github.com/apache/tomcat-connectors/blob/main/native/apache-2.
>> 0/mod_jk.c#L2954#L2973
>> I did some more testing:
>>
>> Variant 1: JkMount /demo/* ajp13_worker;use_server_errors=401
>> Variant 2: JkMount /demo/* ajp13_worker
>>
>> ignoring what variant 2 changes for regular request: reading the
>> source comment my understanding is, that for both variants a HEAD
>> request (by definition must have an empty response body) should let
>> Apache httpd handle the error code.
>>
>> But the return code for jk_handler looks different:
>>
>> Variant 1: s.http_response_status
>> Variant 2: r->status
>
> Although this looks different on the first glance it seems to be the same.
>
>> The response only seems correct for variant 1 - which is configured
>> to let Apache httpd handle all responses for status codes >= 401. For
>> variant 2 mod_jk seems to handle the response itself - contrary to
>> what the comment explains.
>
> This leads to the next assumption, that whenever there is a special handling
> for use_server_errors there should be something similar for the case with an
> empty/non-existing response body.
>
> There is
> https://github.com/apache/tomcat-connectors/blob/main/native/common/jk
> _ajp_common.c#L1991#L1993 with no corresponding (pseudo) code like
>
> if (!r->something_like_bodyct && r->http_response_status >=
> JK_HTTP_BAD_REQUEST){
> r->response_blocked = JK_TRUE;
> }
>
> Adding code like this (sorry, i could not find out how to determine if there
> is a response body) fixes the issue with the wrong response body for a HEAD
> request. But we miss the WWW-Authenticate header now.
>
> Digging further we find
> https://github.com/apache/tomcat-connectors/blob/main/native/apache-2.
> 0/mod_jk.c#L331#L353 which has a special treatment for 401 HTTP
> Unauthorized. But again, only for use_server_errors.
>
> This should be fixable by extending the condition like this
>
> if ((s->extension.use_server_error_pages &&
> status >= s->extension.use_server_error_pages) ||
> (!r->sent_bodyct && r->status >= HTTP_BAD_REQUEST)) {
>
>
> But the WWW-Authenticate header is still missing. So i'm wrong, again.
> Although it feels like i'm close.
>
>> Am 12.02.2022 um 14:24 schrieb Stefan Mayr:
>>> Hello Tomcat users,
>>>
>>> this week we were debugging a strange connection issue which I
>>> tracked down to an interference between Apache httpd and mod_jk.
>>>
>>> For the full picture, the infrastructure setup contains
>>>
>>> 1. a Loadbalancer providing HTTPS, HTTP/1.1 and HTTP/2 for Clients.
>>> 2. an Apache httpd 2.4 webserver (HTTP only) with mod_jk 3. a Tomcat
>>> mit AJP-Connector
>>>
>>> We have an application doing many different HEAD requests against an
>>> application running in the Tomcat server. The requests contain an
>>> Authorization header for Basic authentication. Expected response is
>>> a HTTP 200 OK or HTTP 401 if this particular user is not allowed to
>>> access that ressource. Because this is a HEAD request there must not
>>> be a response body according to RFC 2616.
>>>
>>> If there is a response body in the response to the HEAD request our
>>> loadbalancer does strange things: aborts the connection if the
>>> clients uses HTTP/2, SSL errors if using HTTP/1.1. But this is an
>>> issue on its own which we might have to solve with the vendor.
>>>
>>> Now comes the point where I need your help. We have a httpd
>>> configuration with mod_jk which generates these invalid response
>>> bodies on HEAD requests. I have a gut feeling this could be a bug
>>> with mod_jk.
>>>
>>> For demonstration purpose i created a minimal demo app which only is
>>> a WEB-INF/web.xml >> xmlns="https://jakarta.ee/xml/ns/jakartaee;
>>> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
>>> xsi:schemaLocation="https://jakarta.ee/xml/ns/jakartaee
>>>
>>>
AW: mod_jk interference with ErrorDocument/Alias on HEAD request
Hello Stefan,
by spec / RFC, a HEAD request is not allowed to return any body.
Greetings,
Thomas
-Ursprüngliche Nachricht-
Von: Stefan Mayr
Gesendet: Montag, 14. Februar 2022 23:07
An: users@tomcat.apache.org
Betreff: Re: mod_jk interference with ErrorDocument/Alias on HEAD request
Hello again,
a self-compiled mod_jk 1.2.48 shows the same issue.
Am 13.02.2022 um 18:37 schrieb Stefan Mayr:
> Hi,
>
> looking at the source code
> https://github.com/apache/tomcat-connectors/blob/main/native/apache-2.
> 0/mod_jk.c#L2954#L2973
> I did some more testing:
>
> Variant 1: JkMount /demo/* ajp13_worker;use_server_errors=401
> Variant 2: JkMount /demo/* ajp13_worker
>
> ignoring what variant 2 changes for regular request: reading the
> source comment my understanding is, that for both variants a HEAD
> request (by definition must have an empty response body) should let
> Apache httpd handle the error code.
>
> But the return code for jk_handler looks different:
>
> Variant 1: s.http_response_status
> Variant 2: r->status
Although this looks different on the first glance it seems to be the same.
> The response only seems correct for variant 1 - which is configured to
> let Apache httpd handle all responses for status codes >= 401. For
> variant 2 mod_jk seems to handle the response itself - contrary to
> what the comment explains.
This leads to the next assumption, that whenever there is a special handling
for use_server_errors there should be something similar for the case with an
empty/non-existing response body.
There is
https://github.com/apache/tomcat-connectors/blob/main/native/common/jk_ajp_common.c#L1991#L1993
with no corresponding (pseudo) code like
if (!r->something_like_bodyct && r->http_response_status >=
JK_HTTP_BAD_REQUEST){
r->response_blocked = JK_TRUE;
}
Adding code like this (sorry, i could not find out how to determine if there is
a response body) fixes the issue with the wrong response body for a HEAD
request. But we miss the WWW-Authenticate header now.
Digging further we find
https://github.com/apache/tomcat-connectors/blob/main/native/apache-2.0/mod_jk.c#L331#L353
which has a special treatment for 401 HTTP Unauthorized. But again, only for
use_server_errors.
This should be fixable by extending the condition like this
if ((s->extension.use_server_error_pages &&
status >= s->extension.use_server_error_pages) ||
(!r->sent_bodyct && r->status >= HTTP_BAD_REQUEST)) {
But the WWW-Authenticate header is still missing. So i'm wrong, again.
Although it feels like i'm close.
> Am 12.02.2022 um 14:24 schrieb Stefan Mayr:
>> Hello Tomcat users,
>>
>> this week we were debugging a strange connection issue which I
>> tracked down to an interference between Apache httpd and mod_jk.
>>
>> For the full picture, the infrastructure setup contains
>>
>> 1. a Loadbalancer providing HTTPS, HTTP/1.1 and HTTP/2 for Clients.
>> 2. an Apache httpd 2.4 webserver (HTTP only) with mod_jk 3. a Tomcat
>> mit AJP-Connector
>>
>> We have an application doing many different HEAD requests against an
>> application running in the Tomcat server. The requests contain an
>> Authorization header for Basic authentication. Expected response is a
>> HTTP 200 OK or HTTP 401 if this particular user is not allowed to
>> access that ressource. Because this is a HEAD request there must not
>> be a response body according to RFC 2616.
>>
>> If there is a response body in the response to the HEAD request our
>> loadbalancer does strange things: aborts the connection if the
>> clients uses HTTP/2, SSL errors if using HTTP/1.1. But this is an
>> issue on its own which we might have to solve with the vendor.
>>
>> Now comes the point where I need your help. We have a httpd
>> configuration with mod_jk which generates these invalid response
>> bodies on HEAD requests. I have a gut feeling this could be a bug
>> with mod_jk.
>>
>> For demonstration purpose i created a minimal demo app which only is
>> a WEB-INF/web.xml > xmlns="https://jakarta.ee/xml/ns/jakartaee;
>> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
>> xsi:schemaLocation="https://jakarta.ee/xml/ns/jakartaee
>>
>> https://jakarta.ee/xml/ns/jakartaee/web-app_5_0.xsd;
>> version="5.0">
>>
>>
>> Login
>> /*
>>
>>
>> manager
>>
>>
>>
>> manager
>>
>>
>> BASIC
>>
>>
>>
>> Then I place a JkMount in my Apache httpd configuration (+ minimal
>> worker.properties):
>>
>> JkMount /demo/* ajp13_worker
>>
>> Testing this with curl works like expected:
>>
>> root@1ae8973f1b6b:~# curl -I -v localhost/demo/
>> * Trying 127.0.0.1:80...
>> * TCP_NODELAY set
>> * Connected to localhost (127.0.0.1)
AW: mod_jk interference with ErrorDocument/Alias on HEAD request
Hello, the spec https://datatracker.ietf.org/doc/html/rfc7231#page-25 says in chapter 4.3.2: " The HEAD method is identical to GET except that the server MUST NOT send a message body in the response (i.e., the response terminates at the end of the header section)." Greetings, Thomas -Ursprüngliche Nachricht- Von: Stefan Mayr Gesendet: Sonntag, 13. Februar 2022 18:37 An: users@tomcat.apache.org Betreff: Re: mod_jk interference with ErrorDocument/Alias on HEAD request Hi, looking at the source code https://github.com/apache/tomcat-connectors/blob/main/native/apache-2.0/mod_jk.c#L2954#L2973 I did some more testing: Variant 1: JkMount /demo/* ajp13_worker;use_server_errors=401 Variant 2: JkMount /demo/* ajp13_worker ignoring what variant 2 changes for regular request: reading the source comment my understanding is, that for both variants a HEAD request (by definition must have an empty response body) should let Apache httpd handle the error code. But the return code for jk_handler looks different: Variant 1: s.http_response_status Variant 2: r->status The response only seems correct for variant 1 - which is configured to let Apache httpd handle all responses for status codes >= 401. For variant 2 mod_jk seems to handle the response itself - contrary to what the comment explains. Am 12.02.2022 um 14:24 schrieb Stefan Mayr: > Hello Tomcat users, > > this week we were debugging a strange connection issue which I tracked > down to an interference between Apache httpd and mod_jk. > > For the full picture, the infrastructure setup contains > > 1. a Loadbalancer providing HTTPS, HTTP/1.1 and HTTP/2 for Clients. > 2. an Apache httpd 2.4 webserver (HTTP only) with mod_jk 3. a Tomcat > mit AJP-Connector > > We have an application doing many different HEAD requests against an > application running in the Tomcat server. The requests contain an > Authorization header for Basic authentication. Expected response is a > HTTP 200 OK or HTTP 401 if this particular user is not allowed to > access that ressource. Because this is a HEAD request there must not > be a response body according to RFC 2616. > > If there is a response body in the response to the HEAD request our > loadbalancer does strange things: aborts the connection if the clients > uses HTTP/2, SSL errors if using HTTP/1.1. But this is an issue on its > own which we might have to solve with the vendor. > > Now comes the point where I need your help. We have a httpd > configuration with mod_jk which generates these invalid response > bodies on HEAD requests. I have a gut feeling this could be a bug with mod_jk. > > For demonstration purpose i created a minimal demo app which only is a > WEB-INF/web.xml xmlns="https://jakarta.ee/xml/ns/jakartaee; > xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance; > xsi:schemaLocation="https://jakarta.ee/xml/ns/jakartaee > https://jakarta.ee/xml/ns/jakartaee/web-app_5_0.xsd; > version="5.0"> > > > Login > /* > > > manager > > > > manager > > > BASIC > > > > Then I place a JkMount in my Apache httpd configuration (+ minimal > worker.properties): > > JkMount /demo/* ajp13_worker > > Testing this with curl works like expected: > > root@1ae8973f1b6b:~# curl -I -v localhost/demo/ > * Trying 127.0.0.1:80... > * TCP_NODELAY set > * Connected to localhost (127.0.0.1) port 80 (#0) > HEAD /demo/ > HTTP/1.1 > Host: localhost > User-Agent: curl/7.68.0 > Accept: */* > > > * Mark bundle as not supporting multiuse < HTTP/1.1 401 401 > HTTP/1.1 401 401 > < Date: Sat, 12 Feb 2022 12:57:33 GMT > Date: Sat, 12 Feb 2022 12:57:33 GMT > < Server: Apache/2.4.41 (Ubuntu) > Server: Apache/2.4.41 (Ubuntu) > < Cache-Control: private > Cache-Control: private > < WWW-Authenticate: Basic realm="Authentication required" > WWW-Authenticate: Basic realm="Authentication required" > < Content-Language: en > Content-Language: en > < Content-Type: text/html;charset=utf-8 > Content-Type: text/html;charset=utf-8 > > < > * Connection #0 to host localhost left intact > > But our default setup always includes custom error pages: > > Alias /error/ "/usr/share/apache2/error/" > ErrorDocument 401 /error/HTTP_UNAUTHORIZED.html.var > > If both of those lines are added this results in a response body for > the HEAD request. > > root@1ae8973f1b6b:~# curl -I -v localhost/demo/ > * Trying 127.0.0.1:80... > * TCP_NODELAY set > * Connected to localhost (127.0.0.1) port 80 (#0) > HEAD /demo/ > HTTP/1.1 > Host: localhost > User-Agent: curl/7.68.0 > Accept: */* > > > * Mark bundle as not supporting multiuse < HTTP/1.1 401 401 > HTTP/1.1 401 401 > < Date: Sat, 12 Feb 2022 12:56:27 GMT > Date: Sat, 12 Feb 2022 12:56:27 GMT > < Server:
AW: mod_jk interference with ErrorDocument/Alias on HEAD request
Hello, maybe you can try to set an environment variable which skips interpreting the content-length: https://tomcat.apache.org/connectors-doc/reference/apache.html#Advanced%20Environment%20Variables --> JK_IGNORE_CL To get more information, you can also set the logfile and log-level to debug: JkLogLevel (same reference page as above). Greetings, Thomas -Ursprüngliche Nachricht- Von: Stefan Mayr Gesendet: Samstag, 12. Februar 2022 14:24 An: Tomcat Users List Betreff: mod_jk interference with ErrorDocument/Alias on HEAD request Hello Tomcat users, this week we were debugging a strange connection issue which I tracked down to an interference between Apache httpd and mod_jk. For the full picture, the infrastructure setup contains 1. a Loadbalancer providing HTTPS, HTTP/1.1 and HTTP/2 for Clients. 2. an Apache httpd 2.4 webserver (HTTP only) with mod_jk 3. a Tomcat mit AJP-Connector We have an application doing many different HEAD requests against an application running in the Tomcat server. The requests contain an Authorization header for Basic authentication. Expected response is a HTTP 200 OK or HTTP 401 if this particular user is not allowed to access that ressource. Because this is a HEAD request there must not be a response body according to RFC 2616. If there is a response body in the response to the HEAD request our loadbalancer does strange things: aborts the connection if the clients uses HTTP/2, SSL errors if using HTTP/1.1. But this is an issue on its own which we might have to solve with the vendor. Now comes the point where I need your help. We have a httpd configuration with mod_jk which generates these invalid response bodies on HEAD requests. I have a gut feeling this could be a bug with mod_jk. For demonstration purpose i created a minimal demo app which only is a WEB-INF/web.xml https://jakarta.ee/xml/ns/jakartaee; xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance; xsi:schemaLocation="https://jakarta.ee/xml/ns/jakartaee https://jakarta.ee/xml/ns/jakartaee/web-app_5_0.xsd; version="5.0"> Login /* manager manager BASIC Then I place a JkMount in my Apache httpd configuration (+ minimal worker.properties): JkMount /demo/* ajp13_worker Testing this with curl works like expected: root@1ae8973f1b6b:~# curl -I -v localhost/demo/ * Trying 127.0.0.1:80... * TCP_NODELAY set * Connected to localhost (127.0.0.1) port 80 (#0) > HEAD /demo/ HTTP/1.1 > Host: localhost > User-Agent: curl/7.68.0 > Accept: */* > * Mark bundle as not supporting multiuse < HTTP/1.1 401 401 HTTP/1.1 401 401 < Date: Sat, 12 Feb 2022 12:57:33 GMT Date: Sat, 12 Feb 2022 12:57:33 GMT < Server: Apache/2.4.41 (Ubuntu) Server: Apache/2.4.41 (Ubuntu) < Cache-Control: private Cache-Control: private < WWW-Authenticate: Basic realm="Authentication required" WWW-Authenticate: Basic realm="Authentication required" < Content-Language: en Content-Language: en < Content-Type: text/html;charset=utf-8 Content-Type: text/html;charset=utf-8 < * Connection #0 to host localhost left intact But our default setup always includes custom error pages: Alias /error/ "/usr/share/apache2/error/" ErrorDocument 401 /error/HTTP_UNAUTHORIZED.html.var If both of those lines are added this results in a response body for the HEAD request. root@1ae8973f1b6b:~# curl -I -v localhost/demo/ * Trying 127.0.0.1:80... * TCP_NODELAY set * Connected to localhost (127.0.0.1) port 80 (#0) > HEAD /demo/ HTTP/1.1 > Host: localhost > User-Agent: curl/7.68.0 > Accept: */* > * Mark bundle as not supporting multiuse < HTTP/1.1 401 401 HTTP/1.1 401 401 < Date: Sat, 12 Feb 2022 12:56:27 GMT Date: Sat, 12 Feb 2022 12:56:27 GMT < Server: Apache/2.4.41 (Ubuntu) Server: Apache/2.4.41 (Ubuntu) < Cache-Control: private Cache-Control: private < WWW-Authenticate: Basic realm="Authentication required" WWW-Authenticate: Basic realm="Authentication required" < Content-Language: en Content-Language: en < Content-Type: text/html;charset=utf-8 Content-Type: text/html;charset=utf-8 < * Excess found: excess = 589 url = /demo/ (zero-length body) * Connection #0 to host localhost left intact Checking with tcpdump on port 8009 we see the expected response without a body from the Tomcat AJP connector. The tcpdump von port 80 reveals httpd is adding the configured ErrorDocument as response body. If we comment out either the Alias or ErrorDocument directive the response is correct again. Doing similar tests with CGI/PHP applications always show the correct response without a response body. This only affects requests which use mod_jk. So far I could reproduce this on SLES12 SP5 and SLES15 SP3 running Apache httpd 2.4.51 and a self compile mod_jk 1.2.46 (same with
