JNI memory leak?
For the past few months we’ve been trying to trace what looks like gradual memory creep. After some long-running experiments it seems due to memory leaking when jni_invoke_static(JNIEnv_*, JavaValue*, _jobject*, JNICallType, _jmethodID*, JNI_ArgumentPusher*, Thread*) is invoked. Somewhere. My environment is Tomcat running a proxy webapp. It does TLS termination, authentication and then forwards the call to local services. It doesn’t do much else, it’s a relatively small application. Some (possibly relevant) versions and config parameters: Tomcat 8.5 Java 8u241 (Oracle) Heap size = 360Mb MAX_ALLOC_ARENA=2 MALLOC_TRIM_THRESHOLD_=250048 jdk.nio.maxCachedBufferSize=25600 We couldn’t find any proof of memory leaking on the Java side. When we turn on NativeMemoryTracking=detail and we take a snapshot shortly after starting, we see (just one block shown): [0x03530e462f9a] JNIHandleBlock::allocate_block(Thread*)+0xaa [0x03530e3f759a] JavaCallWrapper::JavaCallWrapper(methodHandle, Handle, JavaValue*, Thread*)+0x6a [0x03530e3fa000] JavaCalls::call_helper(JavaValue*, methodHandle*, JavaCallArguments*, Thread*)+0x8f0 [0x03530e4454a1] jni_invoke_static(JNIEnv_*, JavaValue*, _jobject*, JNICallType, _jmethodID*, JNI_ArgumentPusher*, Thread*) [clone .isra.96] [clone .constprop.117]+0x1e1 (malloc=33783KB type=Internal #110876) Then we run it under heavy load for a few weeks and take another snapshot: [0x03530e462f9a] JNIHandleBlock::allocate_block(Thread*)+0xaa [0x03530e3f759a] JavaCallWrapper::JavaCallWrapper(methodHandle, Handle, JavaValue*, Thread*)+0x6a [0x03530e3fa000] JavaCalls::call_helper(JavaValue*, methodHandle*, JavaCallArguments*, Thread*)+0x8f0 [0x03530e4454a1] jni_invoke_static(JNIEnv_*, JavaValue*, _jobject*, JNICallType, _jmethodID*, JNI_ArgumentPusher*, Thread*) [clone .isra.96] [clone .constprop.117]+0x1e1 (malloc=726749KB type=Internal #2385226) While other blocks also show some variation, none show growth like this one. When I do some math on the number (726749KB - 33783KB) / (2385226 – 110876) it comes down to a pretty even 312 bytes per allocation. And we leaked just under 700Mb. While not immediately problematic, this does not bode well for our customers who run this service for months. I’d like to avoid telling them they need to restart this service every two weeks to reclaim memory. Has anyone seen something like this? Any way it could be avoided? Mark Boon
Re: Fwd: Tomcat 9.0.31 - BUG
On 03/04/2020 19:04, sathis kumar wrote:
> *Tomcat Users List >, *
> *"Thans for your reply.. When i tried with simple JSP", "worked. Fine..
> When i tried deploying my other application", "ran into that issue :("*
That points to an application problem, not a Tomcat problem.
You need to remove things from the application until you have the
simplest possible test case that reproduces the problem. And I expect
that, along the way, you'll find that you are packaging a different
(buggy) version of the EL API in your application - and that is what is
causing the problem.
Mark
>
>
>
> javax.servlet.ServletException: javax.el.ELException: Unable to find
> ExpressionFactory of type: # Licensed to the Apache Software
> Foundation (ASF) under one or more
> org.apache.jsp.index_jsp._jspService(index_jsp.java:428)
> org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)
> javax.servlet.http.HttpServlet.service(HttpServlet.java:741)
>
> org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:477)
> org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:385)
> org.apache.jasper.servlet.JspServlet.service(JspServlet.java:329)
> javax.servlet.http.HttpServlet.service(HttpServlet.java:741)
> org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)
>
> *ype* Exception Report
> *Message* javax.servlet.ServletException: javax.el.ELException: Unable to
> find ExpressionFactory of type: # Licensed to the Apache Software
> Foundation (ASF) under one or more
> *Description* The server encountered an unexpected condition that prevented
> it from fulfilling the request.
> *Exception*
>
> org.apache.jasper.JasperException: javax.servlet.ServletException:
> javax.el.ELException: Unable to find ExpressionFactory of type: #
> Licensed to the Apache Software Foundation (ASF) under one or more
>
> org.apache.jasper.servlet.JspServletWrapper.handleJspException(JspServletWrapper.java:639)
>
> org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:500)
> org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:385)
> org.apache.jasper.servlet.JspServlet.service(JspServlet.java:329)
> javax.servlet.http.HttpServlet.service(HttpServlet.java:741)
> org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)
>
>
> On Fri, Apr 3, 2020 at 11:44 AM Mark Thomas wrote:
>
>> On 03/04/2020 16:33, sathis kumar wrote:
>>> Thanks much for your response. I am using the el-api.jar coming along
>> with
>>> Tomcat9.0.31/lib - Still a sample problem..
>>> JDK version we are using is - > 1.8
>>
>>
>> Please provide the simplest possible set of steps to recreate this error
>> from a clean Apache Tomcat 9.0.31 install.
>>
>> It should be possible to do this by adding a single JSP to a clean
>> install. Please provide us with that JSP.
>>
>> Mark
>>
>>
>>>
>>>
>>> On Fri, Apr 3, 2020 at 11:16 AM Christopher Schultz <
>>> ch...@christopherschultz.net> wrote:
>>>
>>> Sathis,
>>>
>>> You have run across this known bug:
>>> https://bz.apache.org/bugzilla/show_bug.cgi?id=64097
>>>
>>> Please read that bug report for more details, particularly comment #8.
>>>
>>> Hope that helps,
>>> -chris
>>>
>>> On 4/3/20 11:05, sathis kumar wrote:
>> -- Forwarded message - From: sathis kumar
>> Date: Fri, Apr 3, 2020 at 9:38 AM Subject:
>> Tomcat 9.0.31 - BUG To:
>>
>>
>> My Application is working fine in prev version of all tomcat,
>> Planning to upgrade to 9.0.31 . I am running into the following
>> issue.. Pls help
>>
>> org.apache.jasper.JasperException:
>> org.apache.jasper.JasperException: Unable to compile class for JSP
>> at
>> org.apache.jasper.servlet.JspServletWrapper.handleJspException(JspServ
>>> letWrapper.java:605)
>>
>>
>>> at
>> org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.
>>> java:423)
>>
>>
>>> Caused by: javax.el.ELException: Unable to find ExpressionFactory of typ
>>> e:
>> # Licensed to the Apache Software Foundation (ASF) under one or
>> more at
>> javax.el.ExpressionFactory.newInstance(ExpressionFactory.java:152)
>> at javax.el.ExpressionFactory.newInstance(ExpressionFactory.java:9
>> any idea on this .. in older version 9.0.19 applications are
>> working fine
>>
>>
>>
-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
>>>
>>
>>
>> -
>> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
>> For additional commands, e-mail: users-h...@tomcat.apache.org
>>
>>
>
-
To unsubscribe, e-mail:
Re: Fwd: Tomcat 9.0.31 - BUG
*Tomcat Users List >, *
*"Thans for your reply.. When i tried with simple JSP", "worked. Fine..
When i tried deploying my other application", "ran into that issue :("*
javax.servlet.ServletException: javax.el.ELException: Unable to find
ExpressionFactory of type: # Licensed to the Apache Software
Foundation (ASF) under one or more
org.apache.jsp.index_jsp._jspService(index_jsp.java:428)
org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)
javax.servlet.http.HttpServlet.service(HttpServlet.java:741)
org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:477)
org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:385)
org.apache.jasper.servlet.JspServlet.service(JspServlet.java:329)
javax.servlet.http.HttpServlet.service(HttpServlet.java:741)
org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)
*ype* Exception Report
*Message* javax.servlet.ServletException: javax.el.ELException: Unable to
find ExpressionFactory of type: # Licensed to the Apache Software
Foundation (ASF) under one or more
*Description* The server encountered an unexpected condition that prevented
it from fulfilling the request.
*Exception*
org.apache.jasper.JasperException: javax.servlet.ServletException:
javax.el.ELException: Unable to find ExpressionFactory of type: #
Licensed to the Apache Software Foundation (ASF) under one or more
org.apache.jasper.servlet.JspServletWrapper.handleJspException(JspServletWrapper.java:639)
org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:500)
org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:385)
org.apache.jasper.servlet.JspServlet.service(JspServlet.java:329)
javax.servlet.http.HttpServlet.service(HttpServlet.java:741)
org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)
On Fri, Apr 3, 2020 at 11:44 AM Mark Thomas wrote:
> On 03/04/2020 16:33, sathis kumar wrote:
> > Thanks much for your response. I am using the el-api.jar coming along
> with
> > Tomcat9.0.31/lib - Still a sample problem..
> > JDK version we are using is - > 1.8
>
>
> Please provide the simplest possible set of steps to recreate this error
> from a clean Apache Tomcat 9.0.31 install.
>
> It should be possible to do this by adding a single JSP to a clean
> install. Please provide us with that JSP.
>
> Mark
>
>
> >
> >
> > On Fri, Apr 3, 2020 at 11:16 AM Christopher Schultz <
> > ch...@christopherschultz.net> wrote:
> >
> > Sathis,
> >
> > You have run across this known bug:
> > https://bz.apache.org/bugzilla/show_bug.cgi?id=64097
> >
> > Please read that bug report for more details, particularly comment #8.
> >
> > Hope that helps,
> > -chris
> >
> > On 4/3/20 11:05, sathis kumar wrote:
> -- Forwarded message - From: sathis kumar
> Date: Fri, Apr 3, 2020 at 9:38 AM Subject:
> Tomcat 9.0.31 - BUG To:
>
>
> My Application is working fine in prev version of all tomcat,
> Planning to upgrade to 9.0.31 . I am running into the following
> issue.. Pls help
>
> org.apache.jasper.JasperException:
> org.apache.jasper.JasperException: Unable to compile class for JSP
> at
> org.apache.jasper.servlet.JspServletWrapper.handleJspException(JspServ
> > letWrapper.java:605)
>
>
> > at
> org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.
> > java:423)
>
>
> > Caused by: javax.el.ELException: Unable to find ExpressionFactory of typ
> > e:
> # Licensed to the Apache Software Foundation (ASF) under one or
> more at
> javax.el.ExpressionFactory.newInstance(ExpressionFactory.java:152)
> at javax.el.ExpressionFactory.newInstance(ExpressionFactory.java:9
> any idea on this .. in older version 9.0.19 applications are
> working fine
>
>
>
> >>
> >> -
> >> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> >> For additional commands, e-mail: users-h...@tomcat.apache.org
> >>
> >>
> >
>
>
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>
>
--
Cheers,
Sathis Kumar
Re: Fwd: Tomcat 9.0.31 - BUG
On 03/04/2020 16:33, sathis kumar wrote: > Thanks much for your response. I am using the el-api.jar coming along with > Tomcat9.0.31/lib - Still a sample problem.. > JDK version we are using is - > 1.8 Please provide the simplest possible set of steps to recreate this error from a clean Apache Tomcat 9.0.31 install. It should be possible to do this by adding a single JSP to a clean install. Please provide us with that JSP. Mark > > > On Fri, Apr 3, 2020 at 11:16 AM Christopher Schultz < > ch...@christopherschultz.net> wrote: > > Sathis, > > You have run across this known bug: > https://bz.apache.org/bugzilla/show_bug.cgi?id=64097 > > Please read that bug report for more details, particularly comment #8. > > Hope that helps, > -chris > > On 4/3/20 11:05, sathis kumar wrote: -- Forwarded message - From: sathis kumar Date: Fri, Apr 3, 2020 at 9:38 AM Subject: Tomcat 9.0.31 - BUG To: My Application is working fine in prev version of all tomcat, Planning to upgrade to 9.0.31 . I am running into the following issue.. Pls help org.apache.jasper.JasperException: org.apache.jasper.JasperException: Unable to compile class for JSP at org.apache.jasper.servlet.JspServletWrapper.handleJspException(JspServ > letWrapper.java:605) > at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper. > java:423) > Caused by: javax.el.ELException: Unable to find ExpressionFactory of typ > e: # Licensed to the Apache Software Foundation (ASF) under one or more at javax.el.ExpressionFactory.newInstance(ExpressionFactory.java:152) at javax.el.ExpressionFactory.newInstance(ExpressionFactory.java:9 any idea on this .. in older version 9.0.19 applications are working fine >> >> - >> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org >> For additional commands, e-mail: users-h...@tomcat.apache.org >> >> > - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Fwd: Tomcat 9.0.31 - BUG
Thanks much for your response. I am using the el-api.jar coming along with Tomcat9.0.31/lib - Still a sample problem.. JDK version we are using is - > 1.8 On Fri, Apr 3, 2020 at 11:16 AM Christopher Schultz < ch...@christopherschultz.net> wrote: > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA256 > > Sathis, > > You have run across this known bug: > https://bz.apache.org/bugzilla/show_bug.cgi?id=64097 > > Please read that bug report for more details, particularly comment #8. > > Hope that helps, > - -chris > > On 4/3/20 11:05, sathis kumar wrote: > > -- Forwarded message - From: sathis kumar > > Date: Fri, Apr 3, 2020 at 9:38 AM Subject: > > Tomcat 9.0.31 - BUG To: > > > > > > My Application is working fine in prev version of all tomcat, > > Planning to upgrade to 9.0.31 . I am running into the following > > issue.. Pls help > > > > org.apache.jasper.JasperException: > > org.apache.jasper.JasperException: Unable to compile class for JSP > > at > > org.apache.jasper.servlet.JspServletWrapper.handleJspException(JspServ > letWrapper.java:605) > > > > > at > > org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper. > java:423) > > > > > Caused by: javax.el.ELException: Unable to find ExpressionFactory of typ > e: > > # Licensed to the Apache Software Foundation (ASF) under one or > > more at > > javax.el.ExpressionFactory.newInstance(ExpressionFactory.java:152) > > at javax.el.ExpressionFactory.newInstance(ExpressionFactory.java:9 > > any idea on this .. in older version 9.0.19 applications are > > working fine > > > > > > > -BEGIN PGP SIGNATURE- > Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/ > > iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAl6HUzgACgkQHPApP6U8 > pFiKcRAAxKaby2xj6kibPXp4hc2j9s0bF09e89wLKDtn7/q0Vr45M7pFKLikl041 > sjvM1sNw+YmycRDmoNp8iUTG9BMZ5n2HZI/MpCwr8z13MSmAMuitY8eaJ9sF5WjI > Lus/G/FU7oC5r3ln6+tlKFolpAlyg/dTdM5JHEwXnM+0AD4pljMXLvFzzsOVdA5b > n0a7+VDBmkz1KfubofXOnCXkie58WQ/c9z4CKTo6peZGCYlItEXJ8NVF1WQxKFg3 > dDThrJJvMtthzzgau0WJJ5PUhEXgLuarpavPZIxZ6vRHcZGLzq7Ed15Xz+WEtOPS > 2hmkX987SvKGPWqTcwsKvcNsIfiGC99EP+svSlTvDOe/qa+FHdZEKyWW+5M6HBHz > 1jWqaaUWHClHZ/AkbDIRAC4RoShQmOL7g2XOXgMVNkj0vAq24T+66u/HHj+/6vXf > KBSXU0SuiMM8zLoTdB3YKOgNj2sML4xtyKvjOyksThrMdzJ5rTXOC4YcIFQI6jlg > lYMAzc+dWDdjBSXIlFqqiiNAXTlrfLCBRZKUMLYa5XVtfbyv7hz4l2LCoPmo2nDf > baZDg7rhvhJTV9ytk+GikU186F4jpAYiY2Osduh327WsooR9W/0Gw76iOL17/IDH > a63/Itv/JN65hwJWeKEas8wBVmvK2UM8ua/bjnuf8AD63fQ1Grc= > =NX9x > -END PGP SIGNATURE- > > - > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > > -- Cheers, Sathis Kumar
Re: Fwd: Tomcat 9.0.31 - BUG
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Sathis, You have run across this known bug: https://bz.apache.org/bugzilla/show_bug.cgi?id=64097 Please read that bug report for more details, particularly comment #8. Hope that helps, - -chris On 4/3/20 11:05, sathis kumar wrote: > -- Forwarded message - From: sathis kumar > Date: Fri, Apr 3, 2020 at 9:38 AM Subject: > Tomcat 9.0.31 - BUG To: > > > My Application is working fine in prev version of all tomcat, > Planning to upgrade to 9.0.31 . I am running into the following > issue.. Pls help > > org.apache.jasper.JasperException: > org.apache.jasper.JasperException: Unable to compile class for JSP > at > org.apache.jasper.servlet.JspServletWrapper.handleJspException(JspServ letWrapper.java:605) > > at > org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper. java:423) > > Caused by: javax.el.ELException: Unable to find ExpressionFactory of typ e: > # Licensed to the Apache Software Foundation (ASF) under one or > more at > javax.el.ExpressionFactory.newInstance(ExpressionFactory.java:152) > at javax.el.ExpressionFactory.newInstance(ExpressionFactory.java:9 > any idea on this .. in older version 9.0.19 applications are > working fine > > > -BEGIN PGP SIGNATURE- Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/ iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAl6HUzgACgkQHPApP6U8 pFiKcRAAxKaby2xj6kibPXp4hc2j9s0bF09e89wLKDtn7/q0Vr45M7pFKLikl041 sjvM1sNw+YmycRDmoNp8iUTG9BMZ5n2HZI/MpCwr8z13MSmAMuitY8eaJ9sF5WjI Lus/G/FU7oC5r3ln6+tlKFolpAlyg/dTdM5JHEwXnM+0AD4pljMXLvFzzsOVdA5b n0a7+VDBmkz1KfubofXOnCXkie58WQ/c9z4CKTo6peZGCYlItEXJ8NVF1WQxKFg3 dDThrJJvMtthzzgau0WJJ5PUhEXgLuarpavPZIxZ6vRHcZGLzq7Ed15Xz+WEtOPS 2hmkX987SvKGPWqTcwsKvcNsIfiGC99EP+svSlTvDOe/qa+FHdZEKyWW+5M6HBHz 1jWqaaUWHClHZ/AkbDIRAC4RoShQmOL7g2XOXgMVNkj0vAq24T+66u/HHj+/6vXf KBSXU0SuiMM8zLoTdB3YKOgNj2sML4xtyKvjOyksThrMdzJ5rTXOC4YcIFQI6jlg lYMAzc+dWDdjBSXIlFqqiiNAXTlrfLCBRZKUMLYa5XVtfbyv7hz4l2LCoPmo2nDf baZDg7rhvhJTV9ytk+GikU186F4jpAYiY2Osduh327WsooR9W/0Gw76iOL17/IDH a63/Itv/JN65hwJWeKEas8wBVmvK2UM8ua/bjnuf8AD63fQ1Grc= =NX9x -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Fwd: Tomcat 9.0.31 - BUG
-- Forwarded message - From: sathis kumar Date: Fri, Apr 3, 2020 at 9:38 AM Subject: Tomcat 9.0.31 - BUG To: My Application is working fine in prev version of all tomcat, Planning to upgrade to 9.0.31 . I am running into the following issue.. Pls help org.apache.jasper.JasperException: org.apache.jasper.JasperException: Unable to compile class for JSP at org.apache.jasper.servlet.JspServletWrapper.handleJspException(JspServletWrapper.java:605) at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:423) Caused by: javax.el.ELException: Unable to find ExpressionFactory of type: # Licensed to the Apache Software Foundation (ASF) under one or more at javax.el.ExpressionFactory.newInstance(ExpressionFactory.java:152) at javax.el.ExpressionFactory.newInstance(ExpressionFactory.java:9 any idea on this .. in older version 9.0.19 applications are working fine -- Cheers, Sathis Kumar -- Cheers, Sathis Kumar
Re: tomcat7 does not log x-forwarded-for
Hello Mark,
I could test the change today, unfortunately only with moderate
success. The change to %a or %h, unfortunately did not work, but the
readout with the header value per %{X-Forwarded-For}i worked. The log
doesn't look nice, but we get the value. Nevertheless I am still
looking for a nicer solution. I was really hoping on %a that we would
only see the IP of the load balancer if no header is set, but still
always see an IP.
Do you have any more ideas what to watch?
PS:For a Tomcat update I have to ask other contacts if it is easy to
do.
--
Frank Tornack
Am Mittwoch, den 01.04.2020, 09:35 +0100 schrieb Mark Thomas:
> On 01/04/2020 06:56, Frank Tornack wrote:
> > Am Dienstag, den 31.03.2020, 15:47 +0100 schrieb Mark Thomas:
> > > On 31/03/2020 07:26, Frank Tornack wrote:
>
>
>
> > > > We used the following configuration for access logging and
> > > > remote
> > > > valve:
> > > > > > > internalProxies="xxx\.xxx\.xxx\.7|xxx\.xxx\.xxx\.8|xxx\.xxx\.xx
> > > > x\.9
> > > > " remoteIpHeader="X-Forwarded-For" />
> > >
> > > The remoteIpHeader setting is unnecessary. That is the default.
> > The way I understand you, it shouldn't bother? I had added it
> > because I
> > was not sure about the capitalization.
>
> Correct. The names of HTTP headers are not case-sensitive. Tomcat
> looks
> them up in a case insensitive manner.
>
> > > > > > > directory="logs"
> > > > prefix="access_log." suffix=".log"
> > > > pattern="%{org.apache.catalina.AccessLog.RemoteAddr}r %l %u %t
> > > > %r %s %b"
> > > > requestAttributesEnabled="true" />
> > >
> > > That isn't correct. You only need to add
> > > requestAttributesEnabled="true"
> > > to the default AccessLogValve. It will handle looking up the
> > > correct
> > > attributes. You don't need to modify the pattern.
> > I already worked with a modified log layout before the change. I
> > only
> > changed the %{org.apache.catalina.AccessLog.RemoteAddr}r. So would
> > I
> > have to replace %{org.apache.catalina.AccessLog.RemoteAddr}r with a
> > %a?
>
> I was using %h but %a should work just as well. Let me check ... yes
> it
> does.
>
> > > I also confirmed that the Valve order doesn't matter.
> > So not like many on the Internet claim it matters?
>
> Many on the internet are often wrong. Or at least the information
> they
> present is incomplete, out of date and/or out of context.
>
> In this specific instance, Valve order is irrelevant.
>
> The general case is a request/response passes through the Valves
> (Engine
> level Valves in order of definition, then Host Valves in order, then
> Context Valves in order), gets processed by the application and then
> returns through the valves in reverse order.
>
> So, generally, order is important. However, there are some
> exceptions.
>
> A Valve can choose not to pass the request/response on to the next
> stage
> but to generate the response itself and return the request/response
> back
> through the Valves it has passed through to that point.
>
> The AccessLogValve is the other exception. It does not generate the
> log
> entry at the point where the request/response passes through it. It
> used
> to a long time ago but that changed with the fix for bug 50582 (in
> 2011).
>
> It is important which Container (Engine, Host or Context) that the
> Valve
> is added to since that will determine which request/response pairs
> get
> logged.
>
> > > > The request header 'X-Forwarded-For'
> > > > was set and load balancer was analyzed via wireshark and
> > > > Request
> > > > Dumper
> > > > Filter. We can see the header in both. However we still only
> > > > get
> > > > the IP
> > > > of the load balancer and not the IP of the remote client. The
> > > > IP
> > > > adresses of the config have already been checked and they are
> > > > correct.
> > >
> > > You can add "... %{X-Forwarded-For}i ..." to the pattern
> > > attribute of
> > > the AccessLog valve to confirm that Tomcat sees this header
> > > value.
> > In the beginning we tested it with Wireshark, later with a
> > RequestDumper in the Tomcat. In both we saw the header.
> >
> > > I've tested this locally with the latest 7.0.x and everything
> > > works
> > > as
> > > expected. I also don't recall any changes / bugs in this area
> > > recently.
> > >
> > > A quick scan of the changelog does identify various improvements
> > > and
> > > fixes since 7.0.76 but none that immediately jump out as likely
> > > to
> > > cause
> > > this simple configuration to fail.
> > Thanks for looking into it.
>
> I also built 7.0.76 and it worked there too. I explicitly set
> internalProxies so my configuration was as close to yours as
> possible.
>
> At this point it looks like configuration error. All I can suggest is
> going back and re-checking everything. Personally, I'd start with a
> clean 7.0.76 install, confirm the headers sent by the proxies and the
> IP
> used by the proxies and then configure the RemoteIpValve.
>
> Mark
>
>
