Re: HttpServletRequest.getRemoteAddr() sometimes returns NULL on Tomcat 9.0.30 and HTTP/2 secure requests
On 31/03/2020 14:36, Manuel Dominguez Sarmiento wrote:
> Thanks, so we would be able to log this with AccessLogValve using the
> following patterns, right?
> %{org.apache.coyote.connectionID}r
> %{org.apache.coyote.streamID}r
Correct.
Mark
>
> *Manuel Dominguez Sarmiento*
>
> On 31/03/2020 10:28, Mark Thomas wrote:
>> On 29/03/2020 16:16, Mark Thomas wrote:
>>> On 29/03/2020 15:58, Manuel Dominguez Sarmiento wrote:
> Do the streams with errors all share the same connection ID? I suspect
> an error on one stream is triggering the closure of the connection and
> what you are seeing is the fallout of closing the connection.
That makes sense and is kind of what we're suspecting. How can we check
the connection/stream id? Can this be logged by AccessLogValve?
>>> Not easily at the moment. That looks like something we should fix. I'll
>>> see what I can do for 9.0.34.
>> From 8.5.54, 9.0.34 and 10.0.0-M4 the following request attributes will
>> be available for HTTP/2
>>
>> org.apache.coyote.connectionID
>> org.apache.coyote.streamID
>>
>> Mark
>>
>> -
>> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
>> For additional commands, e-mail: users-h...@tomcat.apache.org
>>
>
>
-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
Re: HttpServletRequest.getRemoteAddr() sometimes returns NULL on Tomcat 9.0.30 and HTTP/2 secure requests
Thanks, so we would be able to log this with AccessLogValve using the
following patterns, right?
%{org.apache.coyote.connectionID}r
%{org.apache.coyote.streamID}r
*Manuel Dominguez Sarmiento*
On 31/03/2020 10:28, Mark Thomas wrote:
On 29/03/2020 16:16, Mark Thomas wrote:
On 29/03/2020 15:58, Manuel Dominguez Sarmiento wrote:
Do the streams with errors all share the same connection ID? I suspect
an error on one stream is triggering the closure of the connection and
what you are seeing is the fallout of closing the connection.
That makes sense and is kind of what we're suspecting. How can we check
the connection/stream id? Can this be logged by AccessLogValve?
Not easily at the moment. That looks like something we should fix. I'll
see what I can do for 9.0.34.
From 8.5.54, 9.0.34 and 10.0.0-M4 the following request attributes will
be available for HTTP/2
org.apache.coyote.connectionID
org.apache.coyote.streamID
Mark
-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
Re: HttpServletRequest.getRemoteAddr() sometimes returns NULL on Tomcat 9.0.30 and HTTP/2 secure requests
On 29/03/2020 16:16, Mark Thomas wrote: > On 29/03/2020 15:58, Manuel Dominguez Sarmiento wrote: >> >>> Do the streams with errors all share the same connection ID? I suspect >>> an error on one stream is triggering the closure of the connection and >>> what you are seeing is the fallout of closing the connection. >> That makes sense and is kind of what we're suspecting. How can we check >> the connection/stream id? Can this be logged by AccessLogValve? > > Not easily at the moment. That looks like something we should fix. I'll > see what I can do for 9.0.34. >From 8.5.54, 9.0.34 and 10.0.0-M4 the following request attributes will be available for HTTP/2 org.apache.coyote.connectionID org.apache.coyote.streamID Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: HttpServletRequest.getRemoteAddr() sometimes returns NULL on Tomcat 9.0.30 and HTTP/2 secure requests
On 29/03/2020 15:58, Manuel Dominguez Sarmiento wrote: > >> Do the streams with errors all share the same connection ID? I suspect >> an error on one stream is triggering the closure of the connection and >> what you are seeing is the fallout of closing the connection. > That makes sense and is kind of what we're suspecting. How can we check > the connection/stream id? Can this be logged by AccessLogValve? Not easily at the moment. That looks like something we should fix. I'll see what I can do for 9.0.34. Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: HttpServletRequest.getRemoteAddr() sometimes returns NULL on Tomcat 9.0.30 and HTTP/2 secure requests
Do the streams with errors all share the same connection ID? I suspect an error on one stream is triggering the closure of the connection and what you are seeing is the fallout of closing the connection. That makes sense and is kind of what we're suspecting. How can we check the connection/stream id? Can this be logged by AccessLogValve? *Manuel Dominguez Sarmiento *
Re: HttpServletRequest.getRemoteAddr() sometimes returns NULL on Tomcat 9.0.30 and HTTP/2 secure requests
On 28/03/2020 14:00, Manuel Dominguez Sarmiento wrote:
> Hi Mark, we're now on the latest 9.0.33 release and we still see this
> issue intermittently in our logs. Only on HTTP/2 secure requests.
>
> Please see the attached access logs (these represent all the cases for
> one whole day in one single high-volume server).
> Some of the following request fields are NULL (or -1) in these examples:
> - remoteAddr
> - remotePort
> - serverPort
> - requestURI
> - User-Agent
>
> Some requests are missing some of the fields, some of the requests are
> missing others. What is particularly interesting is that the errors are
> clustered around particular timestamps, pointing to some likely issue
> regarding object sharing across several requests.
Do the streams with errors all share the same connection ID? I suspect
an error on one stream is triggering the closure of the connection and
what you are seeing is the fallout of closing the connection.
Mark
>
> Please note that this is not just an issue at the AccessLogValve level.
> These fields contain invalid data while the request is being processed,
> so that is causing unexpected exceptions in our production code. The
> cases are few and isolated, but still this should be looked into.
>
> Any thoughts?
>
> *Manuel Dominguez Sarmiento*
>
> On 05/02/2020 14:12, Manuel Dominguez Sarmiento wrote:
>> Our filter is not doing anything fancy (and it has always worked
>> correctly before we ran into this bug). In pseudo-code:
>>
>> public doFilter(request, response) {
>>
>> String ip = request.getRemoteAddr();
>> boolean isProxy = isProxy(ip);
>> if (isProxy) {
>> String unwrappedIP = unwrapXForwardedFor(request);
>> chain.doFilter(new
>> MobileProxyHidingServletRequestWrapper(request, unwrappedIP), response);
>> } else {
>> chain.doFilter(request, response);
>> }
>> }
>>
>> All that MobileProxyHidingServletRequestWrapper is override
>> getRemoteAddr() returning unwrappedIP instead of delegating to the
>> actual request, while unwrapXForwardedFor() does what the name
>> suggests, which is processing X-Forwarded-For to obtain the
>> originating IP before it hit the detected proxy.
>>
>> *Manuel Dominguez Sarmiento*
>>
>> On 05/02/2020 10:28, Mark Thomas wrote:
>>> On 04/02/2020 22:27, Manuel Dominguez Sarmiento wrote:
We are getting the NPEs in a top-of-the-chain servlet filter which
decorates HttpServletRequest.getRemoteAddr() before actual servlet
processing. Only on HTTP/2 and in a very small number of cases. Perhaps
we should test 9.0.31 and see what happens. When is this new version due
for release?
>>> I'm just working through back-porting some changes and then I'll be
>>> starting the release process. It 9.0.31 should be available towards the
>>> beginning of next week.
>>>
>>> Can you expand on what your filter is doing? When is the call made to
>>> HttpServletRequest.getRemoteAddr() on the original request?
>>>
>>> Mark
>>>
>
>
>
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>
-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
Re: HttpServletRequest.getRemoteAddr() sometimes returns NULL on Tomcat 9.0.30 and HTTP/2 secure requests
Hi Mark, we're now on the latest 9.0.33 release and we still see this
issue intermittently in our logs. Only on HTTP/2 secure requests.
Please see the attached access logs (these represent all the cases for
one whole day in one single high-volume server).
Some of the following request fields are NULL (or -1) in these examples:
- remoteAddr
- remotePort
- serverPort
- requestURI
- User-Agent
Some requests are missing some of the fields, some of the requests are
missing others. What is particularly interesting is that the errors are
clustered around particular timestamps, pointing to some likely issue
regarding object sharing across several requests.
Please note that this is not just an issue at the AccessLogValve level.
These fields contain invalid data while the request is being processed,
so that is causing unexpected exceptions in our production code. The
cases are few and isolated, but still this should be looked into.
Any thoughts?
*Manuel Dominguez Sarmiento*
On 05/02/2020 14:12, Manuel Dominguez Sarmiento wrote:
Our filter is not doing anything fancy (and it has always worked
correctly before we ran into this bug). In pseudo-code:
public doFilter(request, response) {
String ip = request.getRemoteAddr();
boolean isProxy = isProxy(ip);
if (isProxy) {
String unwrappedIP = unwrapXForwardedFor(request);
chain.doFilter(new
MobileProxyHidingServletRequestWrapper(request, unwrappedIP), response);
} else {
chain.doFilter(request, response);
}
}
All that MobileProxyHidingServletRequestWrapper is override
getRemoteAddr() returning unwrappedIP instead of delegating to the
actual request, while unwrapXForwardedFor() does what the name
suggests, which is processing X-Forwarded-For to obtain the
originating IP before it hit the detected proxy.
*Manuel Dominguez Sarmiento*
On 05/02/2020 10:28, Mark Thomas wrote:
On 04/02/2020 22:27, Manuel Dominguez Sarmiento wrote:
We are getting the NPEs in a top-of-the-chain servlet filter which
decorates HttpServletRequest.getRemoteAddr() before actual servlet
processing. Only on HTTP/2 and in a very small number of cases. Perhaps
we should test 9.0.31 and see what happens. When is this new version due
for release?
I'm just working through back-porting some changes and then I'll be
starting the release process. It 9.0.31 should be available towards the
beginning of next week.
Can you expand on what your filter is doing? When is the call made to
HttpServletRequest.getRemoteAddr() on the original request?
Mark
LOOKING FOR ALL ISSUE INSTANCES:
[root@optimus ~]# cat /home/wap/logs/access.2020-03-27.log | grep "^-"
- -1 443 [27/Mar/2020:07:53:12 -0300] "GET /us/en/country.do?method=list
HTTP/2.0" 400 762 "-" "Mozilla/5.0 (Linux; Android 6.0; vivo 1609)
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.93 Mobile Safari/537.36"
- -1 443 [27/Mar/2020:10:48:12 -0300] "GET
/pe/es/subscriptionPlanDetail.do?id=4483=false=2181=46045=true=419634618870==ojo.pe=d=EAIaIQobChMIif6cyOW66AIVKAa5Bh3eRgI6EAEYASAAEgJuRPD_BwE
HTTP/2.0" 400 637 "https://ojo.pe/; "Mozilla/5.0 (Linux; Android 9; LM-X520
Build/PKQ1.190223.001; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0
Chrome/80.0.3987.119 Mobile Safari/537.36 [FB_IAB/FB4A;FBAV/260.0.0.42.118;]"
- -1 443 [27/Mar/2020:14:39:36 -0300] "GET
/cl/es/subscriptionPlanDetail.do?id=4120=false=2131=17450=false=45663=true=380011499904==mobileapp%3A%3A2-com.appstar.callrecorder=EAIaIQobChMI-aa0_8uN6AIVZga5Bh1UBwQ5EAEYASAAEgKAxPD_BwE_cl_smd_ok=32320413=32320413=705b26c82e98b8401b74a463a68180d6=1584044911681=CELLULAR=EFFECTIVE_4G=true
HTTP/2.0" 400 637
"https://wap.renxo.com/cl/es/subscriptionPlanDetail.do?id=4120=false=2131=17450=false=45663=true=380011499904==mobileapp%3A%3A2-com.appstar.callrecorder=EAIaIQobChMI-aa0_8uN6AIVZga5Bh1UBwQ5EAEYASAAEgKAxPD_BwE_cl_smd_ok=32320413=32320413=705b26c82e98b8401b74a463a68180d6=1584044911681=CELLULAR=EFFECTIVE_4G;
"Mozilla/5.0 (Linux; Android 8.1.0; SAMSUNG SM-J710MN Build/M1AJQ)
AppleWebKit/537.36 (KHTML, like Gecko) SamsungBrowser/9.4 Chrome/67.0.3396.87
Mobile Safari/537.36"
- -1 443 [27/Mar/2020:17:18:55 -0300] "GET
/ar/es/subscriptionPlanDetail.do?id=4328=16242=2403=48008=true=409370554249=%2Farts%20%26%20entertainment=cuttsite.website=d=EAIaIQobChMI98zj67y76AIVT4p3Ch3riAVXEAEYASAAEgLGavD_BwE
HTTP/2.0" 400 637
Re: [OT] HttpServletRequest.getRemoteAddr() sometimes returns NULL on Tomcat 9.0.30 and HTTP/2 secure requests
Hi Christopher, please see below: 1) The Tomcat valves operate on all webapps. We only need/require this for one particular webapp without affecting the others. Not true; see Konstantin's response. Yes, I realized per-context valve configuration was possible after I sent my original email. 2) The code has been simplified for illustration purposes. Besides X-Forwarded-For, we detect and work around many other custom external mobile proxies which do not use X-Forwarded-For and require custom Geolocation code to detect the ISP and connection type (Google Compression Proxy, Nokia OVI, Novarra, Lotus Flare, Opera Mini, Opera Max, Samsung Max, etc.) - this kind of customization is not possible without custom code. Interesting. Is this something you think would be widely useful Not sure how useful it would be to other users. We are a fairly specialized shop running platforms for mobile carriers, so mobile carrier detection, MSISDN header enrichment, detecting whether users are on a real 3G/4G network or through a modem, hotspot or proxy, etc. are a major concern for us, but I don't believe most users would care about this. and/or would be willing to share with the community? We'd be willing to share our code, however see my comments above. If it's a fast-moving target (e.g. new public proxies are popping-up all the time, or existing proxies keep changing their configuration requirements) then maybe it's not a great fit for a stable product like Tomcat. Most of our proxy detection logic works by checking geolocation data from trusted, paid sources (MaxMind and Digital Element/NetAcuity) so even if the logic is simpe, it requires fresh, updated data from a trusted third-party to be of any use. On the other hand, some of the proxy detection logic instead depends on proprietary header analysis (for instance, Opera Mini inserts many X-OperaMini- prefixed headers). On the other hand, if it could be configured relatively easily (like with a "proxy definitions" file or something), then it could still be very valuable even with a simple or default configuration which only supports some very large proxies (e.g. Akamai, CloudFront, CloudFlare, etc.). This is only useful for small, well-known partner proxies that are supposed to notify us if their IP addresses change. But major proxies change their IP ranges all the time. So, again, this would not really be useful without the third-party geolocation data. This filter is not meant for detecting internal proxies within our control (such as Apache front ends or load balancers), but rather public proxies which are "transparently" (not really) used via some mobile devices and services. Does it matter whether these are "internal" versus "external" proxies? The only real difference is the IP-range of the proxy, right? No, that's not the only difference. Some proxies introduce non-standard behaviour, even if they advertise themselves as "trasparent": - Novarra proxy transcodes HTML for small screens (this proxy is obsolete, but is a prime example of non-transparency) - Opera Mini does all sorts of fancy things to compress responses: it coalesces HTML, images, and JavaScript payloads to minimize the number of streams; modifies HTML to fit devices with small screens and/or low memory/CPU resources, etc. - Chrome Compression Proxy (aka "Data Saver mode") is fairly transparent, but there are many caveats. In particular, MSISDN header enrichment breaks, which is a major issue for services we manage for mobile carriers. See https://developer.chrome.com/multidevice/data-compression-for-isps for more details. - etc. It doesn't matter whether you control the proxy or it's an external service: you still have to secure and validate the connection in the same ways, and take the same action(s) on the server-end where you trust the information being presented. See comments above. *Manuel Dominguez Sarmiento*
Re: [OT] HttpServletRequest.getRemoteAddr() sometimes returns NULL on Tomcat 9.0.30 and HTTP/2 secure requests
Hi Konstantin, please see below: You can configure a Valve for a specific web application by placing it into Context configuration for that specific web application (usually that is the "/META-INF/context.xml" configuration file). [1] [1] http://tomcat.apache.org/tomcat-9.0-doc/config/context.html#Defining_a_context You are correct. I realized this after my original response. We actually already use per-context AccessLogValves. 2. If I understand correctly, the null value from request.getRemoteAddr() means that the client connection has already been closed. Tomcat cannot do much at that point, unless the information has not already been requested (and thus cached) when the connection was still alive. (The recent changes to the AccessLogValve are just that: to request the value earlier.) It is useless to process a request if the connection has already been closed. We request getRemoteAddr() several times during servlet processing (for Geolocation, proxy and carrier detection among other purposes), as well as in top-of-the-chain servlet filters. If the connection has been closed, it must be because of some Tomcat issue. This never happened with earlier Tomcar versions (we started this project many years ago with 6.x). Plus, the issue is only present in a small number of cases on HTTP/2, but not regular HTTPS nor plain HTTP. Why do you say that the null value is an invalid one? I do not see such words in the specification. I disagree. There cannot be a TCP/IP, HTTP, HTTPS, HTTP/2 or HTTP/3-QUIC connection without an originating IP, so it makes no sense to return NULL under any circumstances. Furthermore, the spec Javadoc for getRemoteAddr() states "Returns: a String containing the IP address of the client that sent the request" Contrast this to getRemoteUser() for instance, which states: "Returns: a String specifying the login of the user making this request, *or null if the user login is not known*" If NULL were a valid return value, it would be explicitly mentioned. 3. Just as a note (I would not recommend it for your specific use case) One known way to detect a closed connection is to trigger parameter parsing and look whether an error flag (implemented as an attribute of a Request) was set by it. See the implementation of org.apache.catalina.filters.FailedRequestFilter for an example. As a workaround, since we only have a very small number of requests that trigger this, we have implemented a filter that discards those requests and returns HTTP 400 "Bad Request", until the issue is resolved. 4. Do you run with the following configuration setting turned on? org.apache.catalina.connector.RECYCLE_FACADES=true We do not use this option, so we must be running with the default="false" See https://cwiki.apache.org/confluence/x/yColBg#TroubleshootingandDiagnostics-TroubleshootingunexpectedResponsestateproblems Thanks, I'll read through this. *Manuel Dominguez Sarmiento*
Re: [OT] HttpServletRequest.getRemoteAddr() sometimes returns NULL on Tomcat 9.0.30 and HTTP/2 secure requests
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256
Manuel,
On 2/5/20 1:29 PM, Manuel Dominguez Sarmiento wrote:
> Yes, there are two reasons:
>
> 1) The Tomcat valves operate on all webapps. We only need/require
> this for one particular webapp without affecting the others.
Not true; see Konstantin's response.
> 2) The code has been simplified for illustration purposes. Besides
> X-Forwarded-For, we detect and work around many other custom
> external mobile proxies which do not use X-Forwarded-For and
> require custom Geolocation code to detect the ISP and connection
> type (Google Compression Proxy, Nokia OVI, Novarra, Lotus Flare,
> Opera Mini, Opera Max, Samsung Max, etc.) - this kind of
> customization is not possible without custom code.
Interesting. Is this something you think would be widely useful and/or
would be willing to share with the community? If it's a fast-moving
target (e.g. new public proxies are popping-up all the time, or
existing proxies keep changing their configuration requirements) then
maybe it's not a great fit for a stable product like Tomcat.
On the other hand, if it could be configured relatively easily (like
with a "proxy definitions" file or something), then it could still be
very valuable even with a simple or default configuration which only
supports some very large proxies (e.g. Akamai, CloudFront, CloudFlare,
etc.).
> This filter is not meant for detecting internal proxies within our
> control (such as Apache front ends or load balancers), but rather
> public proxies which are "transparently" (not really) used via
> some mobile devices and services.
Does it matter whether these are "internal" versus "external" proxies?
The only real difference is the IP-range of the proxy, right? It
doesn't matter whether you control the proxy or it's an external
service: you still have to secure and validate the connection in the
same ways, and take the same action(s) on the server-end where you
trust the information being presented.
- -chris
> On 2/5/20 12:12 PM, Manuel Dominguez Sarmiento wrote:
Our filter is not doing anything fancy (and it has always
worked correctly before we ran into this bug). In
pseudo-code:
public doFilter(request, response) {
String ip = request.getRemoteAddr(); boolean isProxy =
isProxy(ip); if (isProxy) { String unwrappedIP =
unwrapXForwardedFor(request); chain.doFilter(new
MobileProxyHidingServletRequestWrapper(request,
unwrappedIP), response); } else { chain.doFilter(request,
response); } }
All that MobileProxyHidingServletRequestWrapper is override
getRemoteAddr() returning unwrappedIP instead of delegating
to the actual request, while unwrapXForwardedFor() does what
the name suggests, which is processing X-Forwarded-For to
obtain the originating IP before it hit the detected proxy.
> Any reason not to use the valves Tomcat provides to do pretty much
> this exact thing?
>
> https://tomcat.apache.org/tomcat-9.0-doc/config/valve.html#Remote_IP_V
al
>
>
ve
>
> -chris
>>
>> -
>>
>>
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
>> For additional commands, e-mail: users-h...@tomcat.apache.org
>>
>
-BEGIN PGP SIGNATURE-
Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/
iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAl48MBsACgkQHPApP6U8
pFgaXw/+P0rEqNNqW2oM3Yajrzk3mgYt41kuq2zGrjdr/6vBGR1fUmrx92krOkM/
1bzFOkbWnRWIktFmjyARaGK12F3/zczvU7JRmRazEKXraxJ0vgPg8NPy8YN4KhBG
efptbwqbbdJ02r1i8eVdNjacYBd5/gT51qFRaaTseIz16prSUxaT4RXoui2je68v
fkA7pl5jAND8B7Nr6uczGVQwQELWTEwNKUiz6ji+GAwKF7oZMCX64p7TYEcD7o2p
BgcO5VzyjwdcXuDTOJ4RqTgaHc09aqdP+VRiWV18RGr0rYTISq6zW6lHXT9Goc0m
qH5mKp402aiX03rK8n9F+523K52X7xR7B0+48r+4UcPDrWiWDzuF/IVI93ugNLw5
ITexNxhvmhvC2PHY5CRoa69Us2YG9iZo0z579RIFuYT4/75DR3+oakqIAHz2FU5C
x+n4Qkbaj/v/O56Ja983bKbWul5XCTLnL2AmLKfcnK7er1CpWx9elN7oXvhiFYC1
8ceLtQU1A36vElFWUj1jC82M41tuQdcdL3VADcvi6bANpAYEsEG/RbNnBu/0WgfD
ByxN40M9Km+wtzDlxqW5Sg2eKm4tdNk2duqtTv9+NPH4H+5Tu9+OQ/r5gTPrsJfn
knEd9/Los4QCrAYSsHF6PXD/bbdlJbUpRdIMYUKJXA6BNMT2LNo=
=1ts9
-END PGP SIGNATURE-
-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
Re: [OT] HttpServletRequest.getRemoteAddr() sometimes returns NULL on Tomcat 9.0.30 and HTTP/2 secure requests
ср, 5 февр. 2020 г. в 21:29, Manuel Dominguez Sarmiento : > > Yes, there are two reasons: > > 1) The Tomcat valves operate on all webapps. We only need/require this > for one particular webapp without affecting the others. You can configure a Valve for a specific web application by placing it into Context configuration for that specific web application (usually that is the "/META-INF/context.xml" configuration file). [1] [1] http://tomcat.apache.org/tomcat-9.0-doc/config/context.html#Defining_a_context 2. If I understand correctly, the null value from request.getRemoteAddr() means that the client connection has already been closed. Tomcat cannot do much at that point, unless the information has not already been requested (and thus cached) when the connection was still alive. (The recent changes to the AccessLogValve are just that: to request the value earlier.) It is useless to process a request if the connection has already been closed. Why do you say that the null value is an invalid one? I do not see such words in the specification. 3. Just as a note (I would not recommend it for your specific use case) One known way to detect a closed connection is to trigger parameter parsing and look whether an error flag (implemented as an attribute of a Request) was set by it. See the implementation of org.apache.catalina.filters.FailedRequestFilter for an example. 4. Do you run with the following configuration setting turned on? org.apache.catalina.connector.RECYCLE_FACADES=true See https://cwiki.apache.org/confluence/x/yColBg#TroubleshootingandDiagnostics-TroubleshootingunexpectedResponsestateproblems Best regards, Konstantin Kolinko - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: [OT] HttpServletRequest.getRemoteAddr() sometimes returns NULL on Tomcat 9.0.30 and HTTP/2 secure requests
Yes, there are two reasons:
1) The Tomcat valves operate on all webapps. We only need/require this
for one particular webapp without affecting the others.
2) The code has been simplified for illustration purposes. Besides
X-Forwarded-For, we detect and work around many other custom external
mobile proxies which do not use X-Forwarded-For and require custom
Geolocation code to detect the ISP and connection type (Google
Compression Proxy, Nokia OVI, Novarra, Lotus Flare, Opera Mini, Opera
Max, Samsung Max, etc.) - this kind of customization is not possible
without custom code. This filter is not meant for detecting internal
proxies within our control (such as Apache front ends or load
balancers), but rather public proxies which are "transparently" (not
really) used via some mobile devices and services.
*
Manuel Dominguez Sarmiento*
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256
Manuel,
On 2/5/20 12:12 PM, Manuel Dominguez Sarmiento wrote:
Our filter is not doing anything fancy (and it has always worked
correctly before we ran into this bug). In pseudo-code:
public doFilter(request, response) {
String ip = request.getRemoteAddr(); boolean isProxy =
isProxy(ip); if (isProxy) { String unwrappedIP =
unwrapXForwardedFor(request); chain.doFilter(new
MobileProxyHidingServletRequestWrapper(request, unwrappedIP),
response); } else { chain.doFilter(request, response); } }
All that MobileProxyHidingServletRequestWrapper is override
getRemoteAddr() returning unwrappedIP instead of delegating to the
actual request, while unwrapXForwardedFor() does what the name
suggests, which is processing X-Forwarded-For to obtain the
originating IP before it hit the detected proxy.
Any reason not to use the valves Tomcat provides to do pretty much
this exact thing?
https://tomcat.apache.org/tomcat-9.0-doc/config/valve.html#Remote_IP_Val
ve
- -chris
-BEGIN PGP SIGNATURE-
Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/
iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAl47AfoACgkQHPApP6U8
pFgywBAAxq4KZaxhO/X0EaoA3G3TDKURbDSrA2Hbke0QVgP3tX6QX6MDMEkbqTuv
x4S1xgXeNgWNIE9Dy62t4ciC6VZ21v6Rl4JMTOmtUO0ES+OBNTSYXstA1ZywBoMD
bRXHNjjdPtxAhhqQblfpKter+vn/k/PZ+45GgZSLNMrrJR+1idAafWs7YwpCHxfL
zh+C5C3uHu+bznCiOF/NQQ5d3/0mHyOj5nSpYx2ImSH+rgj0/Ch5w43L4chD1Lqj
aw2sqqC2z4Fzp3R8uNX/uMRV4flck9KILrfOe00BrMGVfPVuDDGa2J4NhSTs0oNx
Pda5j19QAP6wh6vIKH5KVM8fUWcb5KthcT34cxKttFl3gL0EedE40y97PHKxihJ5
BaT9M3MD0Lsh/mtgrSGcxHU1G0Vs5hHw9e9Zm0XqgQeLsuIALnSHcFyQcGBY05sH
9jljk/l+iMIp9xi3fXyCgYHRpCvbwOuT92V4pMYmwLCzvpfIrVspTi+7AxzbipBV
7tTvH4om0PSchj+Gj3ayOkQZsAyDZbJXbV4izxN3p1c8AJT5ZUmJnYqpKNtfAJpW
w/3OxIXoTUq3bbpb6AHrJe36+QU6+/C2WNqLBcevElr6T1oObKfzy1CjoUHxUQCm
wWxrSFac6atc56mVuPYAjs1DZ0sKpvts/ih6AwDF8+0jBzeJX+g=
=5GpE
-END PGP SIGNATURE-
-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
Re: [OT] HttpServletRequest.getRemoteAddr() sometimes returns NULL on Tomcat 9.0.30 and HTTP/2 secure requests
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256
Manuel,
On 2/5/20 12:12 PM, Manuel Dominguez Sarmiento wrote:
> Our filter is not doing anything fancy (and it has always worked
> correctly before we ran into this bug). In pseudo-code:
>
> public doFilter(request, response) {
>
> String ip = request.getRemoteAddr(); boolean isProxy =
> isProxy(ip); if (isProxy) { String unwrappedIP =
> unwrapXForwardedFor(request); chain.doFilter(new
> MobileProxyHidingServletRequestWrapper(request, unwrappedIP),
> response); } else { chain.doFilter(request, response); } }
>
> All that MobileProxyHidingServletRequestWrapper is override
> getRemoteAddr() returning unwrappedIP instead of delegating to the
> actual request, while unwrapXForwardedFor() does what the name
> suggests, which is processing X-Forwarded-For to obtain the
> originating IP before it hit the detected proxy.
Any reason not to use the valves Tomcat provides to do pretty much
this exact thing?
https://tomcat.apache.org/tomcat-9.0-doc/config/valve.html#Remote_IP_Val
ve
- -chris
-BEGIN PGP SIGNATURE-
Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/
iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAl47AfoACgkQHPApP6U8
pFgywBAAxq4KZaxhO/X0EaoA3G3TDKURbDSrA2Hbke0QVgP3tX6QX6MDMEkbqTuv
x4S1xgXeNgWNIE9Dy62t4ciC6VZ21v6Rl4JMTOmtUO0ES+OBNTSYXstA1ZywBoMD
bRXHNjjdPtxAhhqQblfpKter+vn/k/PZ+45GgZSLNMrrJR+1idAafWs7YwpCHxfL
zh+C5C3uHu+bznCiOF/NQQ5d3/0mHyOj5nSpYx2ImSH+rgj0/Ch5w43L4chD1Lqj
aw2sqqC2z4Fzp3R8uNX/uMRV4flck9KILrfOe00BrMGVfPVuDDGa2J4NhSTs0oNx
Pda5j19QAP6wh6vIKH5KVM8fUWcb5KthcT34cxKttFl3gL0EedE40y97PHKxihJ5
BaT9M3MD0Lsh/mtgrSGcxHU1G0Vs5hHw9e9Zm0XqgQeLsuIALnSHcFyQcGBY05sH
9jljk/l+iMIp9xi3fXyCgYHRpCvbwOuT92V4pMYmwLCzvpfIrVspTi+7AxzbipBV
7tTvH4om0PSchj+Gj3ayOkQZsAyDZbJXbV4izxN3p1c8AJT5ZUmJnYqpKNtfAJpW
w/3OxIXoTUq3bbpb6AHrJe36+QU6+/C2WNqLBcevElr6T1oObKfzy1CjoUHxUQCm
wWxrSFac6atc56mVuPYAjs1DZ0sKpvts/ih6AwDF8+0jBzeJX+g=
=5GpE
-END PGP SIGNATURE-
-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
Re: HttpServletRequest.getRemoteAddr() sometimes returns NULL on Tomcat 9.0.30 and HTTP/2 secure requests
Our filter is not doing anything fancy (and it has always worked
correctly before we ran into this bug). In pseudo-code:
public doFilter(request, response) {
String ip = request.getRemoteAddr();
boolean isProxy = isProxy(ip);
if (isProxy) {
String unwrappedIP = unwrapXForwardedFor(request);
chain.doFilter(new
MobileProxyHidingServletRequestWrapper(request, unwrappedIP), response);
} else {
chain.doFilter(request, response);
}
}
All that MobileProxyHidingServletRequestWrapper is override
getRemoteAddr() returning unwrappedIP instead of delegating to the
actual request, while unwrapXForwardedFor() does what the name suggests,
which is processing X-Forwarded-For to obtain the originating IP before
it hit the detected proxy.
*Manuel Dominguez Sarmiento*
On 05/02/2020 10:28, Mark Thomas wrote:
On 04/02/2020 22:27, Manuel Dominguez Sarmiento wrote:
We are getting the NPEs in a top-of-the-chain servlet filter which
decorates HttpServletRequest.getRemoteAddr() before actual servlet
processing. Only on HTTP/2 and in a very small number of cases. Perhaps
we should test 9.0.31 and see what happens. When is this new version due
for release?
I'm just working through back-porting some changes and then I'll be
starting the release process. It 9.0.31 should be available towards the
beginning of next week.
Can you expand on what your filter is doing? When is the call made to
HttpServletRequest.getRemoteAddr() on the original request?
Mark
Re: HttpServletRequest.getRemoteAddr() sometimes returns NULL on Tomcat 9.0.30 and HTTP/2 secure requests
On 04/02/2020 22:27, Manuel Dominguez Sarmiento wrote: > We are getting the NPEs in a top-of-the-chain servlet filter which > decorates HttpServletRequest.getRemoteAddr() before actual servlet > processing. Only on HTTP/2 and in a very small number of cases. Perhaps > we should test 9.0.31 and see what happens. When is this new version due > for release? I'm just working through back-porting some changes and then I'll be starting the release process. It 9.0.31 should be available towards the beginning of next week. Can you expand on what your filter is doing? When is the call made to HttpServletRequest.getRemoteAddr() on the original request? Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: HttpServletRequest.getRemoteAddr() sometimes returns NULL on Tomcat 9.0.30 and HTTP/2 secure requests
We are getting the NPEs in a top-of-the-chain servlet filter which decorates HttpServletRequest.getRemoteAddr() before actual servlet processing. Only on HTTP/2 and in a very small number of cases. Perhaps we should test 9.0.31 and see what happens. When is this new version due for release? *- **Manuel Dominguez Sarmiento* Thanks Mark. I just wanted to clarify that the issue is not only present when the request arrives at AccessLogValve, but while the request is being serviced as well. We noticed this bug because we were getting random NullPointerExceptions when trying to do anything with the result from HttpServletRequest.getRemoteAddr() in our servlets. Should the fix solve this as well? It depends. The only way I can see this happening is if the application swallows an IOException and continues. It probably won't fix that and I don't think we'd consider an NPE an error in those circumstances. Mark
Re: HttpServletRequest.getRemoteAddr() sometimes returns NULL on Tomcat 9.0.30 and HTTP/2 secure requests
On 04/02/2020 15:37, Manuel Dominguez Sarmiento wrote: > Thanks Mark. I just wanted to clarify that the issue is not only present > when the request arrives at AccessLogValve, but while the request is > being serviced as well. > We noticed this bug because we were getting random NullPointerExceptions > when trying to do anything with the result from > HttpServletRequest.getRemoteAddr() in our servlets. > > Should the fix solve this as well? It depends. The only way I can see this happening is if the application swallows an IOException and continues. It probably won't fix that and I don't think we'd consider an NPE an error in those circumstances. Mark > > - Manuel Dominguez Sarmiento >> On 03/02/2020 19:24, Mark Thomas wrote: >> >> >> >>> I haven't fixed this but I can reproduce it easily with the h2spec test >>> suite. As I have a reproducible test case I'm hopeful I'll be able to >>> find a fix fairly quickly. >> I've just committed a fix. It is in: >> - master for 10.0.0.0-M1 onwards >> - 9.0.x for 9.0.31 onwards >> - 8.5.x for 8.5.51 onwards >> >> 7.0.x was not affected. >> >> The issue was that by the time the AccessLogValve tried to obtain the >> remote host/addr/port the connection had been closed and the information >> was not available. This was a result of the change below which removed a >> potential path for a request to see incorrect remote client information. >> >> This entry in the changelog: >> >> Decouple the socket wrapper, which is not recycled, from the NIOx >> channel after close, and replace it with a dummy static object. >> >> >> Mark >> >> - >> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org >> For additional commands, e-mail: users-h...@tomcat.apache.org >> > > > - > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: HttpServletRequest.getRemoteAddr() sometimes returns NULL on Tomcat 9.0.30 and HTTP/2 secure requests
Thanks Mark. I just wanted to clarify that the issue is not only present when the request arrives at AccessLogValve, but while the request is being serviced as well. We noticed this bug because we were getting random NullPointerExceptions when trying to do anything with the result from HttpServletRequest.getRemoteAddr() in our servlets. Should the fix solve this as well? - Manuel Dominguez Sarmiento On 03/02/2020 19:24, Mark Thomas wrote: I haven't fixed this but I can reproduce it easily with the h2spec test suite. As I have a reproducible test case I'm hopeful I'll be able to find a fix fairly quickly. I've just committed a fix. It is in: - master for 10.0.0.0-M1 onwards - 9.0.x for 9.0.31 onwards - 8.5.x for 8.5.51 onwards 7.0.x was not affected. The issue was that by the time the AccessLogValve tried to obtain the remote host/addr/port the connection had been closed and the information was not available. This was a result of the change below which removed a potential path for a request to see incorrect remote client information. This entry in the changelog: Decouple the socket wrapper, which is not recycled, from the NIOx channel after close, and replace it with a dummy static object. Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: HttpServletRequest.getRemoteAddr() sometimes returns NULL on Tomcat 9.0.30 and HTTP/2 secure requests
On 03/02/2020 19:24, Mark Thomas wrote: > I haven't fixed this but I can reproduce it easily with the h2spec test > suite. As I have a reproducible test case I'm hopeful I'll be able to > find a fix fairly quickly. I've just committed a fix. It is in: - master for 10.0.0.0-M1 onwards - 9.0.x for 9.0.31 onwards - 8.5.x for 8.5.51 onwards 7.0.x was not affected. The issue was that by the time the AccessLogValve tried to obtain the remote host/addr/port the connection had been closed and the information was not available. This was a result of the change below which removed a potential path for a request to see incorrect remote client information. This entry in the changelog: Decouple the socket wrapper, which is not recycled, from the NIOx channel after close, and replace it with a dummy static object. Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: HttpServletRequest.getRemoteAddr() sometimes returns NULL on Tomcat 9.0.30 and HTTP/2 secure requests
On 24/01/2020 12:26, Manuel Dominguez Sarmiento wrote: > Hi Mark, thanks for your feedback. Please see below: >> On 23/01/2020 13:40, Manuel Dominguez Sarmiento wrote: >>> Hi, we started >>> noticing that HttpServletRequest.getRemoteAddr() was >>> sometimes returning NULL (which is invalid according to the servlet >>> spec), about 20-50 times per day (we have high-load servers which >>> routinely handle over 100 requests per second). This only happens on >>> secure HTTP/2 requests. >>> Our server.xml is below (confidential data has been modified). We are >>> using the NIO connector, Tomcat Native + APR libraries (but NOT the APR >>> connector). The useAsyncIO flag is disabled because of server lockups >>> we've experienced with this flag enabled on our production servers. Any >>> ideas? >> Nothing comes to mind immediately. >> >> Mark > Please let me know if there's any diagnostic info that might help with > this and we'll produce it. Sorry it has taken me so long to get back to this. I haven't fixed this but I can reproduce it easily with the h2spec test suite. As I have a reproducible test case I'm hopeful I'll be able to find a fix fairly quickly. Kind regards, Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: HttpServletRequest.getRemoteAddr() sometimes returns NULL on Tomcat 9.0.30 and HTTP/2 secure requests
Hi Mark, thanks for your feedback. Please see below: On 23/01/2020 13:40, Manuel Dominguez Sarmiento wrote:> Hi, we started noticing that HttpServletRequest.getRemoteAddr() was sometimes returning NULL (which is invalid according to the servlet spec), about 20-50 times per day (we have high-load servers which routinely handle over 100 requests per second). This only happens on secure HTTP/2 requests. We noticed this while running the latest 9.0.30, but looking at our logs for the past few months, this was also happening on 9.0.21 as well as 9.0.22 which are the previous versions we had deployed. This issue was not present in 9.0.17, which was the version we were using before 9.0.21. Before anyone cries out "there is a bug in your code", Tomcat itself reports a NULL remoteAddr via AccessLogValve in these cases. It is also interesting that the remotePort is reported as "-1". We looked through the changelog, but we can't figure out what change might have triggered this (presumably some change between 9.0.17 and 9.0.21). It looks like some kind of async race condition during the HTTP/2 upgrade, but this is only an educated guess. I'll take a look. You mention async. Are these requests using the Servlet async API? No, we're not using the async API in any capacity. Just the regular servlet API: doGet() / doPost() / etc. Perhaps my usage of "async" was confusing (as that can refer to several different things in Tomcat). I just meant that it feels like a race condition between different threads. Our server.xml is below (confidential data has been modified). We are using the NIO connector, Tomcat Native + APR libraries (but NOT the APR connector). The useAsyncIO flag is disabled because of server lockups we've experienced with this flag enabled on our production servers. Any ideas? Nothing comes to mind immediately. Mark Please let me know if there's any diagnostic info that might help with this and we'll produce it. Thanks again, - Manuel Dominguez Sarmiento
Re: HttpServletRequest.getRemoteAddr() sometimes returns NULL on Tomcat 9.0.30 and HTTP/2 secure requests
On 23/01/2020 13:40, Manuel Dominguez Sarmiento wrote:> Hi, we started noticing that HttpServletRequest.getRemoteAddr() was > sometimes returning NULL (which is invalid according to the servlet > spec), about 20-50 times per day (we have high-load servers which > routinely handle over 100 requests per second). This only happens on > secure HTTP/2 requests. > > We noticed this while running the latest 9.0.30, but looking at our logs > for the past few months, this was also happening on 9.0.21 as well as > 9.0.22 which are the previous versions we had deployed. This issue was > not present in 9.0.17, which was the version we were using before 9.0.21. > > Before anyone cries out "there is a bug in your code", Tomcat itself > reports a NULL remoteAddr via AccessLogValve in these cases. It is also > interesting that the remotePort is reported as "-1". > > We looked through the changelog, but we can't figure out what change > might have triggered this (presumably some change between 9.0.17 and > 9.0.21). It looks like some kind of async race condition during the > HTTP/2 upgrade, but this is only an educated guess. I'll take a look. You mention async. Are these requests using the Servlet async API? > Our server.xml is below (confidential data has been modified). We are > using the NIO connector, Tomcat Native + APR libraries (but NOT the APR > connector). The useAsyncIO flag is disabled because of server lockups > we've experienced with this flag enabled on our production servers. Any > ideas? Nothing comes to mind immediately. Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
HttpServletRequest.getRemoteAddr() sometimes returns NULL on Tomcat 9.0.30 and HTTP/2 secure requests
Hi, we started noticing that HttpServletRequest.getRemoteAddr() was sometimes returning NULL (which is invalid according to the servlet spec), about 20-50 times per day (we have high-load servers which routinely handle over 100 requests per second). This only happens on secure HTTP/2 requests. We noticed this while running the latest 9.0.30, but looking at our logs for the past few months, this was also happening on 9.0.21 as well as 9.0.22 which are the previous versions we had deployed. This issue was not present in 9.0.17, which was the version we were using before 9.0.21. Before anyone cries out "there is a bug in your code", Tomcat itself reports a NULL remoteAddr via AccessLogValve in these cases. It is also interesting that the remotePort is reported as "-1". We looked through the changelog, but we can't figure out what change might have triggered this (presumably some change between 9.0.17 and 9.0.21). It looks like some kind of async race condition during the HTTP/2 upgrade, but this is only an educated guess. Our server.xml is below (confidential data has been modified). We are using the NIO connector, Tomcat Native + APR libraries (but NOT the APR connector). The useAsyncIO flag is disabled because of server lockups we've experienced with this flag enabled on our production servers. Any ideas? - Manuel Dominguez Sarmiento
