Re: [OT] HttpServletRequest.getRemoteAddr() sometimes returns NULL on Tomcat 9.0.30 and HTTP/2 secure requests
Hi Christopher, please see below: 1) The Tomcat valves operate on all webapps. We only need/require this for one particular webapp without affecting the others. Not true; see Konstantin's response. Yes, I realized per-context valve configuration was possible after I sent my original email. 2) The code has been simplified for illustration purposes. Besides X-Forwarded-For, we detect and work around many other custom external mobile proxies which do not use X-Forwarded-For and require custom Geolocation code to detect the ISP and connection type (Google Compression Proxy, Nokia OVI, Novarra, Lotus Flare, Opera Mini, Opera Max, Samsung Max, etc.) - this kind of customization is not possible without custom code. Interesting. Is this something you think would be widely useful Not sure how useful it would be to other users. We are a fairly specialized shop running platforms for mobile carriers, so mobile carrier detection, MSISDN header enrichment, detecting whether users are on a real 3G/4G network or through a modem, hotspot or proxy, etc. are a major concern for us, but I don't believe most users would care about this. and/or would be willing to share with the community? We'd be willing to share our code, however see my comments above. If it's a fast-moving target (e.g. new public proxies are popping-up all the time, or existing proxies keep changing their configuration requirements) then maybe it's not a great fit for a stable product like Tomcat. Most of our proxy detection logic works by checking geolocation data from trusted, paid sources (MaxMind and Digital Element/NetAcuity) so even if the logic is simpe, it requires fresh, updated data from a trusted third-party to be of any use. On the other hand, some of the proxy detection logic instead depends on proprietary header analysis (for instance, Opera Mini inserts many X-OperaMini- prefixed headers). On the other hand, if it could be configured relatively easily (like with a "proxy definitions" file or something), then it could still be very valuable even with a simple or default configuration which only supports some very large proxies (e.g. Akamai, CloudFront, CloudFlare, etc.). This is only useful for small, well-known partner proxies that are supposed to notify us if their IP addresses change. But major proxies change their IP ranges all the time. So, again, this would not really be useful without the third-party geolocation data. This filter is not meant for detecting internal proxies within our control (such as Apache front ends or load balancers), but rather public proxies which are "transparently" (not really) used via some mobile devices and services. Does it matter whether these are "internal" versus "external" proxies? The only real difference is the IP-range of the proxy, right? No, that's not the only difference. Some proxies introduce non-standard behaviour, even if they advertise themselves as "trasparent": - Novarra proxy transcodes HTML for small screens (this proxy is obsolete, but is a prime example of non-transparency) - Opera Mini does all sorts of fancy things to compress responses: it coalesces HTML, images, and JavaScript payloads to minimize the number of streams; modifies HTML to fit devices with small screens and/or low memory/CPU resources, etc. - Chrome Compression Proxy (aka "Data Saver mode") is fairly transparent, but there are many caveats. In particular, MSISDN header enrichment breaks, which is a major issue for services we manage for mobile carriers. See https://developer.chrome.com/multidevice/data-compression-for-isps for more details. - etc. It doesn't matter whether you control the proxy or it's an external service: you still have to secure and validate the connection in the same ways, and take the same action(s) on the server-end where you trust the information being presented. See comments above. *Manuel Dominguez Sarmiento*
Re: [OT] HttpServletRequest.getRemoteAddr() sometimes returns NULL on Tomcat 9.0.30 and HTTP/2 secure requests
Hi Konstantin, please see below: You can configure a Valve for a specific web application by placing it into Context configuration for that specific web application (usually that is the "/META-INF/context.xml" configuration file). [1] [1] http://tomcat.apache.org/tomcat-9.0-doc/config/context.html#Defining_a_context You are correct. I realized this after my original response. We actually already use per-context AccessLogValves. 2. If I understand correctly, the null value from request.getRemoteAddr() means that the client connection has already been closed. Tomcat cannot do much at that point, unless the information has not already been requested (and thus cached) when the connection was still alive. (The recent changes to the AccessLogValve are just that: to request the value earlier.) It is useless to process a request if the connection has already been closed. We request getRemoteAddr() several times during servlet processing (for Geolocation, proxy and carrier detection among other purposes), as well as in top-of-the-chain servlet filters. If the connection has been closed, it must be because of some Tomcat issue. This never happened with earlier Tomcar versions (we started this project many years ago with 6.x). Plus, the issue is only present in a small number of cases on HTTP/2, but not regular HTTPS nor plain HTTP. Why do you say that the null value is an invalid one? I do not see such words in the specification. I disagree. There cannot be a TCP/IP, HTTP, HTTPS, HTTP/2 or HTTP/3-QUIC connection without an originating IP, so it makes no sense to return NULL under any circumstances. Furthermore, the spec Javadoc for getRemoteAddr() states "Returns: a String containing the IP address of the client that sent the request" Contrast this to getRemoteUser() for instance, which states: "Returns: a String specifying the login of the user making this request, *or null if the user login is not known*" If NULL were a valid return value, it would be explicitly mentioned. 3. Just as a note (I would not recommend it for your specific use case) One known way to detect a closed connection is to trigger parameter parsing and look whether an error flag (implemented as an attribute of a Request) was set by it. See the implementation of org.apache.catalina.filters.FailedRequestFilter for an example. As a workaround, since we only have a very small number of requests that trigger this, we have implemented a filter that discards those requests and returns HTTP 400 "Bad Request", until the issue is resolved. 4. Do you run with the following configuration setting turned on? org.apache.catalina.connector.RECYCLE_FACADES=true We do not use this option, so we must be running with the default="false" See https://cwiki.apache.org/confluence/x/yColBg#TroubleshootingandDiagnostics-TroubleshootingunexpectedResponsestateproblems Thanks, I'll read through this. *Manuel Dominguez Sarmiento*
Re: [OT] HttpServletRequest.getRemoteAddr() sometimes returns NULL on Tomcat 9.0.30 and HTTP/2 secure requests
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Manuel, On 2/5/20 1:29 PM, Manuel Dominguez Sarmiento wrote: > Yes, there are two reasons: > > 1) The Tomcat valves operate on all webapps. We only need/require > this for one particular webapp without affecting the others. Not true; see Konstantin's response. > 2) The code has been simplified for illustration purposes. Besides > X-Forwarded-For, we detect and work around many other custom > external mobile proxies which do not use X-Forwarded-For and > require custom Geolocation code to detect the ISP and connection > type (Google Compression Proxy, Nokia OVI, Novarra, Lotus Flare, > Opera Mini, Opera Max, Samsung Max, etc.) - this kind of > customization is not possible without custom code. Interesting. Is this something you think would be widely useful and/or would be willing to share with the community? If it's a fast-moving target (e.g. new public proxies are popping-up all the time, or existing proxies keep changing their configuration requirements) then maybe it's not a great fit for a stable product like Tomcat. On the other hand, if it could be configured relatively easily (like with a "proxy definitions" file or something), then it could still be very valuable even with a simple or default configuration which only supports some very large proxies (e.g. Akamai, CloudFront, CloudFlare, etc.). > This filter is not meant for detecting internal proxies within our > control (such as Apache front ends or load balancers), but rather > public proxies which are "transparently" (not really) used via > some mobile devices and services. Does it matter whether these are "internal" versus "external" proxies? The only real difference is the IP-range of the proxy, right? It doesn't matter whether you control the proxy or it's an external service: you still have to secure and validate the connection in the same ways, and take the same action(s) on the server-end where you trust the information being presented. - -chris > On 2/5/20 12:12 PM, Manuel Dominguez Sarmiento wrote: Our filter is not doing anything fancy (and it has always worked correctly before we ran into this bug). In pseudo-code: public doFilter(request, response) { String ip = request.getRemoteAddr(); boolean isProxy = isProxy(ip); if (isProxy) { String unwrappedIP = unwrapXForwardedFor(request); chain.doFilter(new MobileProxyHidingServletRequestWrapper(request, unwrappedIP), response); } else { chain.doFilter(request, response); } } All that MobileProxyHidingServletRequestWrapper is override getRemoteAddr() returning unwrappedIP instead of delegating to the actual request, while unwrapXForwardedFor() does what the name suggests, which is processing X-Forwarded-For to obtain the originating IP before it hit the detected proxy. > Any reason not to use the valves Tomcat provides to do pretty much > this exact thing? > > https://tomcat.apache.org/tomcat-9.0-doc/config/valve.html#Remote_IP_V al > > ve > > -chris >> >> - >> >> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org >> For additional commands, e-mail: users-h...@tomcat.apache.org >> > -BEGIN PGP SIGNATURE- Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/ iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAl48MBsACgkQHPApP6U8 pFgaXw/+P0rEqNNqW2oM3Yajrzk3mgYt41kuq2zGrjdr/6vBGR1fUmrx92krOkM/ 1bzFOkbWnRWIktFmjyARaGK12F3/zczvU7JRmRazEKXraxJ0vgPg8NPy8YN4KhBG efptbwqbbdJ02r1i8eVdNjacYBd5/gT51qFRaaTseIz16prSUxaT4RXoui2je68v fkA7pl5jAND8B7Nr6uczGVQwQELWTEwNKUiz6ji+GAwKF7oZMCX64p7TYEcD7o2p BgcO5VzyjwdcXuDTOJ4RqTgaHc09aqdP+VRiWV18RGr0rYTISq6zW6lHXT9Goc0m qH5mKp402aiX03rK8n9F+523K52X7xR7B0+48r+4UcPDrWiWDzuF/IVI93ugNLw5 ITexNxhvmhvC2PHY5CRoa69Us2YG9iZo0z579RIFuYT4/75DR3+oakqIAHz2FU5C x+n4Qkbaj/v/O56Ja983bKbWul5XCTLnL2AmLKfcnK7er1CpWx9elN7oXvhiFYC1 8ceLtQU1A36vElFWUj1jC82M41tuQdcdL3VADcvi6bANpAYEsEG/RbNnBu/0WgfD ByxN40M9Km+wtzDlxqW5Sg2eKm4tdNk2duqtTv9+NPH4H+5Tu9+OQ/r5gTPrsJfn knEd9/Los4QCrAYSsHF6PXD/bbdlJbUpRdIMYUKJXA6BNMT2LNo= =1ts9 -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: [OT] HttpServletRequest.getRemoteAddr() sometimes returns NULL on Tomcat 9.0.30 and HTTP/2 secure requests
ср, 5 февр. 2020 г. в 21:29, Manuel Dominguez Sarmiento : > > Yes, there are two reasons: > > 1) The Tomcat valves operate on all webapps. We only need/require this > for one particular webapp without affecting the others. You can configure a Valve for a specific web application by placing it into Context configuration for that specific web application (usually that is the "/META-INF/context.xml" configuration file). [1] [1] http://tomcat.apache.org/tomcat-9.0-doc/config/context.html#Defining_a_context 2. If I understand correctly, the null value from request.getRemoteAddr() means that the client connection has already been closed. Tomcat cannot do much at that point, unless the information has not already been requested (and thus cached) when the connection was still alive. (The recent changes to the AccessLogValve are just that: to request the value earlier.) It is useless to process a request if the connection has already been closed. Why do you say that the null value is an invalid one? I do not see such words in the specification. 3. Just as a note (I would not recommend it for your specific use case) One known way to detect a closed connection is to trigger parameter parsing and look whether an error flag (implemented as an attribute of a Request) was set by it. See the implementation of org.apache.catalina.filters.FailedRequestFilter for an example. 4. Do you run with the following configuration setting turned on? org.apache.catalina.connector.RECYCLE_FACADES=true See https://cwiki.apache.org/confluence/x/yColBg#TroubleshootingandDiagnostics-TroubleshootingunexpectedResponsestateproblems Best regards, Konstantin Kolinko - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: [OT] HttpServletRequest.getRemoteAddr() sometimes returns NULL on Tomcat 9.0.30 and HTTP/2 secure requests
Yes, there are two reasons: 1) The Tomcat valves operate on all webapps. We only need/require this for one particular webapp without affecting the others. 2) The code has been simplified for illustration purposes. Besides X-Forwarded-For, we detect and work around many other custom external mobile proxies which do not use X-Forwarded-For and require custom Geolocation code to detect the ISP and connection type (Google Compression Proxy, Nokia OVI, Novarra, Lotus Flare, Opera Mini, Opera Max, Samsung Max, etc.) - this kind of customization is not possible without custom code. This filter is not meant for detecting internal proxies within our control (such as Apache front ends or load balancers), but rather public proxies which are "transparently" (not really) used via some mobile devices and services. * Manuel Dominguez Sarmiento* -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Manuel, On 2/5/20 12:12 PM, Manuel Dominguez Sarmiento wrote: Our filter is not doing anything fancy (and it has always worked correctly before we ran into this bug). In pseudo-code: public doFilter(request, response) { String ip = request.getRemoteAddr(); boolean isProxy = isProxy(ip); if (isProxy) { String unwrappedIP = unwrapXForwardedFor(request); chain.doFilter(new MobileProxyHidingServletRequestWrapper(request, unwrappedIP), response); } else { chain.doFilter(request, response); } } All that MobileProxyHidingServletRequestWrapper is override getRemoteAddr() returning unwrappedIP instead of delegating to the actual request, while unwrapXForwardedFor() does what the name suggests, which is processing X-Forwarded-For to obtain the originating IP before it hit the detected proxy. Any reason not to use the valves Tomcat provides to do pretty much this exact thing? https://tomcat.apache.org/tomcat-9.0-doc/config/valve.html#Remote_IP_Val ve - -chris -BEGIN PGP SIGNATURE- Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/ iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAl47AfoACgkQHPApP6U8 pFgywBAAxq4KZaxhO/X0EaoA3G3TDKURbDSrA2Hbke0QVgP3tX6QX6MDMEkbqTuv x4S1xgXeNgWNIE9Dy62t4ciC6VZ21v6Rl4JMTOmtUO0ES+OBNTSYXstA1ZywBoMD bRXHNjjdPtxAhhqQblfpKter+vn/k/PZ+45GgZSLNMrrJR+1idAafWs7YwpCHxfL zh+C5C3uHu+bznCiOF/NQQ5d3/0mHyOj5nSpYx2ImSH+rgj0/Ch5w43L4chD1Lqj aw2sqqC2z4Fzp3R8uNX/uMRV4flck9KILrfOe00BrMGVfPVuDDGa2J4NhSTs0oNx Pda5j19QAP6wh6vIKH5KVM8fUWcb5KthcT34cxKttFl3gL0EedE40y97PHKxihJ5 BaT9M3MD0Lsh/mtgrSGcxHU1G0Vs5hHw9e9Zm0XqgQeLsuIALnSHcFyQcGBY05sH 9jljk/l+iMIp9xi3fXyCgYHRpCvbwOuT92V4pMYmwLCzvpfIrVspTi+7AxzbipBV 7tTvH4om0PSchj+Gj3ayOkQZsAyDZbJXbV4izxN3p1c8AJT5ZUmJnYqpKNtfAJpW w/3OxIXoTUq3bbpb6AHrJe36+QU6+/C2WNqLBcevElr6T1oObKfzy1CjoUHxUQCm wWxrSFac6atc56mVuPYAjs1DZ0sKpvts/ih6AwDF8+0jBzeJX+g= =5GpE -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: [OT] HttpServletRequest.getRemoteAddr() sometimes returns NULL on Tomcat 9.0.30 and HTTP/2 secure requests
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Manuel, On 2/5/20 12:12 PM, Manuel Dominguez Sarmiento wrote: > Our filter is not doing anything fancy (and it has always worked > correctly before we ran into this bug). In pseudo-code: > > public doFilter(request, response) { > > String ip = request.getRemoteAddr(); boolean isProxy = > isProxy(ip); if (isProxy) { String unwrappedIP = > unwrapXForwardedFor(request); chain.doFilter(new > MobileProxyHidingServletRequestWrapper(request, unwrappedIP), > response); } else { chain.doFilter(request, response); } } > > All that MobileProxyHidingServletRequestWrapper is override > getRemoteAddr() returning unwrappedIP instead of delegating to the > actual request, while unwrapXForwardedFor() does what the name > suggests, which is processing X-Forwarded-For to obtain the > originating IP before it hit the detected proxy. Any reason not to use the valves Tomcat provides to do pretty much this exact thing? https://tomcat.apache.org/tomcat-9.0-doc/config/valve.html#Remote_IP_Val ve - -chris -BEGIN PGP SIGNATURE- Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/ iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAl47AfoACgkQHPApP6U8 pFgywBAAxq4KZaxhO/X0EaoA3G3TDKURbDSrA2Hbke0QVgP3tX6QX6MDMEkbqTuv x4S1xgXeNgWNIE9Dy62t4ciC6VZ21v6Rl4JMTOmtUO0ES+OBNTSYXstA1ZywBoMD bRXHNjjdPtxAhhqQblfpKter+vn/k/PZ+45GgZSLNMrrJR+1idAafWs7YwpCHxfL zh+C5C3uHu+bznCiOF/NQQ5d3/0mHyOj5nSpYx2ImSH+rgj0/Ch5w43L4chD1Lqj aw2sqqC2z4Fzp3R8uNX/uMRV4flck9KILrfOe00BrMGVfPVuDDGa2J4NhSTs0oNx Pda5j19QAP6wh6vIKH5KVM8fUWcb5KthcT34cxKttFl3gL0EedE40y97PHKxihJ5 BaT9M3MD0Lsh/mtgrSGcxHU1G0Vs5hHw9e9Zm0XqgQeLsuIALnSHcFyQcGBY05sH 9jljk/l+iMIp9xi3fXyCgYHRpCvbwOuT92V4pMYmwLCzvpfIrVspTi+7AxzbipBV 7tTvH4om0PSchj+Gj3ayOkQZsAyDZbJXbV4izxN3p1c8AJT5ZUmJnYqpKNtfAJpW w/3OxIXoTUq3bbpb6AHrJe36+QU6+/C2WNqLBcevElr6T1oObKfzy1CjoUHxUQCm wWxrSFac6atc56mVuPYAjs1DZ0sKpvts/ih6AwDF8+0jBzeJX+g= =5GpE -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org