Re: JNI memory leak?
Thanks Chris for taking the time. As you point out, from the threads I can tell we're not using ARP as the names al all starting with "jsse". AFAI could find out BouncyCastle is a pure Java implementation, so that also can't be the cause. Someone suggested PAMLibrary may be the culprit. So I started a thread that makes continuous auth calls to the PAM library. Now there does seems to be an indication memory is leaking very, very slowly. It seems to be roughly in line with the number of auth failures. It looks like PAM throttles auth failures though, hence it's taking such a long time for the evidence to mount. So nothing to see here for this group. Just wanted to give a heads up. Mark On 4/6/20, 12:12 PM, "Christopher Schultz" wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Mark, On 4/3/20 21:48, Mark Boon wrote: > For the past few months we’ve been trying to trace what looks like > gradual memory creep. After some long-running experiments it seems > due to memory leaking when jni_invoke_static(JNIEnv_*, JavaValue*, > _jobject*, JNICallType, _jmethodID*, JNI_ArgumentPusher*, Thread*) > is invoked. Somewhere. > > My environment is Tomcat running a proxy webapp. It does TLS > termination, authentication and then forwards the call to local > services. It doesn’t do much else, it’s a relatively small > application. > > Some (possibly relevant) versions and config parameters: Tomcat > 8.5 Java 8u241 (Oracle) Heap size = 360Mb MAX_ALLOC_ARENA=2 > MALLOC_TRIM_THRESHOLD_=250048 jdk.nio.maxCachedBufferSize=25600 > > We couldn’t find any proof of memory leaking on the Java side. When > we turn on NativeMemoryTracking=detail and we take a snapshot > shortly after starting, we see (just one block shown): > > [0x03530e462f9a] JNIHandleBlock::allocate_block(Thread*)+0xaa > [0x03530e3f759a] JavaCallWrapper::JavaCallWrapper(methodHandle, > Handle, JavaValue*, Thread*)+0x6a [0x03530e3fa000] > JavaCalls::call_helper(JavaValue*, methodHandle*, > JavaCallArguments*, Thread*)+0x8f0 [0x03530e4454a1] > jni_invoke_static(JNIEnv_*, JavaValue*, _jobject*, JNICallType, > _jmethodID*, JNI_ArgumentPusher*, Thread*) [clone .isra.96] [clone > .constprop.117]+0x1e1 (malloc=33783KB type=Internal #110876) > > Then we run it under heavy load for a few weeks and take another > snapshot: > > [0x03530e462f9a] JNIHandleBlock::allocate_block(Thread*)+0xaa > [0x03530e3f759a] JavaCallWrapper::JavaCallWrapper(methodHandle, > Handle, JavaValue*, Thread*)+0x6a [0x03530e3fa000] > JavaCalls::call_helper(JavaValue*, methodHandle*, > JavaCallArguments*, Thread*)+0x8f0 [0x03530e4454a1] > jni_invoke_static(JNIEnv_*, JavaValue*, _jobject*, JNICallType, > _jmethodID*, JNI_ArgumentPusher*, Thread*) [clone .isra.96] [clone > .constprop.117]+0x1e1 (malloc=726749KB type=Internal #2385226) > > While other blocks also show some variation, none show growth like > this one. When I do some math on the number (726749KB - 33783KB) / > (2385226 – 110876) it comes down to a pretty even 312 bytes per > allocation. And we leaked just under 700Mb. While not immediately > problematic, this does not bode well for our customers who run this > service for months. > > I’d like to avoid telling them they need to restart this service > every two weeks to reclaim memory. Has anyone seen something like > this? Any way it could be avoided? That was some very good sleuthing on your part. 312 bytes per allocation will indeed be very difficult to detect unless you are really looking hard for it. On 4/4/20 13:02, Mark Boon wrote: > The connector of the webapp uses Http11NioProtocol. My > understanding is it uses direct-byte-buffers backed by native > memory for the Nio channels. I don't know for sure if that gets > allocated through a JNI call, but that was my assumption. This will definitely use Tomcat's NIO protocol which doesn't use the APR connector. However, you still might be using tcnative to get the crypto engine. Can you confirm the thread-naming convention of your request-processing threads? They will tell you if JSSE or OpenSSL (tcnative) is being used. A few data points: * No Tomcat code directly invokes jni_invoke_static(), but it might do so indirectly through a variety of means. * NIO does use buffers, but those buffers tend to be (a) fairly large - -- on the order of kilobytes -- and (b) re-used for the life of the request-processor thread. It is very possible that there is a very small leak in Tomcat's handling of NIO buffers. I think it's equally likely that there is a bug in the JVM itself. Are you able to try different JVM versions in your test? I would recommend major-version changes,
Re: Catalina PID file
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Konstantin, On 4/24/20 11:27, Konstantin Kolinko wrote: > пт, 24 апр. 2020 г. в 05:25, Christopher Schultz > : >> >> -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 >> >> Rafael, >> >> On 4/23/20 17:18, Rafael Oliveira wrote: >>> I does not happen every time, but it happens, actually it >>> happened twice during a couple of deployments and only in one >>> instance of several that I have. >>> >>> I could not reproduce in a safe and close environment, I got >>> this error in a server with multiple users and multiple tasks >>> and concurred environment. >>> >>> The point is raise a question is this echo $! > "CATALINA_PID" >>> the best approach to get the PID? >> >> It's literally the only way to do it. >> >> $! doesn't get you the PID of some random other process that was >> recently started. It's exactly what it says it is: the PID of the >> last process that was backgrounded by the currently-executing >> script. >> >> If your pid-file contains the wrong PID, it's probably because >> something overwrite overwrote it after it was >> initially-generated. >> >> If you try to start Tomcat twice, for example, like this: >> >> $ CATALINA_HOME/bin/startup.sh [...] $ cat $CATALINA_PID 1234 $ >> CATALINA_HOME/bin/startup.sh [...] $ cat $CATALINA_PID 2345 $ ps >> -ef | grep "catalina.base=$CATALINA_HOME" 1234 >> >> The startup script will always overwrite the PID file because it >> can't tell if it's going to fail (which it will, since the >> shutdown port is already bound to the first-launched instance). >> >> In the case above, the PID-file is clobbered by the second >> process while the first process continues to run (and be the >> correct process id). There isn't much to be done about that >> without resorting to drastic measures, such as having Tomcat >> write its own PID file after it starts up (enough), and getting >> the PID from Java isn't super-straightforward until Java 9. :( > > 1. I do not know what version of Tomcat the OP is using, but the > current catalina.sh has some protection from double runs: > > ps -p $PID >/dev/null 2>&1 if [ $? -eq 0 ] ; then echo "Tomcat > appears to still be running with PID $PID. Start aborted." echo "If > the following process is not a Tomcat process, remove the PID file > and try again:" ps -f -p $PID exit 1 else > > There is some time window between this check and the launch of > Java process (when the PID file is being written out), so there > will always be a race condition here. Yes, but it fails-safe. That is, Tomcat will not start if it thinks another Tomcat is already running. If there is a race, a second attempt to start Tomcat will likely succeed. > 2. If one does not like how Tomcat forks a separate shell process > for a java executable, it is possible to start it with "catalina.sh > run". In this case java is started in the same process (and no PID > file is being written) and all control over this process should be > done in your own code YMMV. > > 3. Tomcat can be configured as a service (daemon) and started with > Apache Commons Daemon executable (jsvc). jsvc knows how to manage > a PID file. See daemon.sh for an example. > >> and getting the PID from Java isn't super-straightforward until >> Java 9. :( > > 4. With our own cat's food: Tomcat-Native has the method: > > org.apache.tomcat.jni.Stdlib.getpid() Interesting, although that does require that tcnative is available. - -chris -BEGIN PGP SIGNATURE- Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/ iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAl6jF6IACgkQHPApP6U8 pFgzhRAApf5Rzn/0n8S2gCKorMlHftPWSo9BUoYkHBAqlMpYzIzVIxvpxujPBC5n P1/ifE+VqK98G6nkEKuJwQI3IZXNvd8zW3x6/rcVKhHG/0d81rYw2AeAxbAt4cUS JqQqwpbInOl0dpcWnkQic2KnjJP+qI25DGL0U3Rsx3J9T8tfQ11Dv9AIlepAO4uu KXgcs2JGlwv6vOzV2MkFuw6/0LkrDV4jlV1uYJFHj9CopO9x1ofNGupu6WjtlqiM FbIuPudmus8tmPP/TZ3lSjuj2Aw2kVq+7vBgsKLAVo/KIXifTP7DXXdiI4J2hcdG su+806PDsuEDwOKyMuWXhKzn2TuE+UVyxYMMTu1VrbX3GlIxPszjswpQgFAvLbNj GEQNarV18E/Ni6Zvw4PM1jJiMtcKyoPxUcVWbBkwDrMcqymPLdWrZEp/lBrwMsw6 XQHLzqHNl8ZUW6YIwd2a0VgycqJzPKVP8zfhvgUzf2NxtMzCNwckebZ4lvhD5S+T cZqCwfR+J8VNOlhB+52JLRl1aotqX5LYMI8HNwQAanfbg5e3940FeBeIZF9XZSb4 +IsVNlBdpS1bsv6yAGGrSk2drM85cHW6pBiEGhqfSfDDlm8xSURNaQLM27GWEQgD T2gbijw711xG3U3bRjoZa445IQ7ZN8phs9mA2n6eDi1WGXwM0Bs= =cdsU -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Catalina PID file
пт, 24 апр. 2020 г. в 05:25, Christopher Schultz : > > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA256 > > Rafael, > > On 4/23/20 17:18, Rafael Oliveira wrote: > > I does not happen every time, but it happens, actually it happened > > twice during a couple of deployments and only in one instance of > > several that I have. > > > > I could not reproduce in a safe and close environment, I got this > > error in a server with multiple users and multiple tasks and > > concurred environment. > > > > The point is raise a question is this echo $! > "CATALINA_PID" the > > best approach to get the PID? > > It's literally the only way to do it. > > $! doesn't get you the PID of some random other process that was > recently started. It's exactly what it says it is: the PID of the last > process that was backgrounded by the currently-executing script. > > If your pid-file contains the wrong PID, it's probably because > something overwrite overwrote it after it was initially-generated. > > If you try to start Tomcat twice, for example, like this: > > $ CATALINA_HOME/bin/startup.sh > [...] > $ cat $CATALINA_PID > 1234 > $ CATALINA_HOME/bin/startup.sh > [...] > $ cat $CATALINA_PID > 2345 > $ ps -ef | grep "catalina.base=$CATALINA_HOME" > 1234 > > The startup script will always overwrite the PID file because it can't > tell if it's going to fail (which it will, since the shutdown port is > already bound to the first-launched instance). > > In the case above, the PID-file is clobbered by the second process > while the first process continues to run (and be the correct process > id). There isn't much to be done about that without resorting to > drastic measures, such as having Tomcat write its own PID file after > it starts up (enough), and getting the PID from Java isn't > super-straightforward until Java 9. :( 1. I do not know what version of Tomcat the OP is using, but the current catalina.sh has some protection from double runs: ps -p $PID >/dev/null 2>&1 if [ $? -eq 0 ] ; then echo "Tomcat appears to still be running with PID $PID. Start aborted." echo "If the following process is not a Tomcat process, remove the PID file and try again:" ps -f -p $PID exit 1 else There is some time window between this check and the launch of Java process (when the PID file is being written out), so there will always be a race condition here. 2. If one does not like how Tomcat forks a separate shell process for a java executable, it is possible to start it with "catalina.sh run". In this case java is started in the same process (and no PID file is being written) and all control over this process should be done in your own code YMMV. 3. Tomcat can be configured as a service (daemon) and started with Apache Commons Daemon executable (jsvc). jsvc knows how to manage a PID file. See daemon.sh for an example. > and getting the PID from Java isn't super-straightforward until Java 9. :( 4. With our own cat's food: Tomcat-Native has the method: org.apache.tomcat.jni.Stdlib.getpid() :) Best regards, Konstantin Kolinko - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat serving old TXT file even after deleting work directory and rebooted
Lame!
It turned out that I have
@WebServlet(name = "AdsTxt", urlPatterns = {"/ads.txt"})
LOL
On Fri, Apr 24, 2020 at 4:48 PM Mladen Adamović
wrote:
> I've changed ads.txt did upload ROOT.war file and it now contains new
> version which is 18892 bytes old, checked files in the filesystem:
>
> root@condor1796 /home/glassfish/apache-tomcat-8.5.5 # find . | grep
> "ads.txt" | xargs ls -l
> -rw-r- 1 glassfish nogroup 18892 Apr 24 2020
> ./appBaseLivingCost/ROOT/ads.txt
> -rw-r- 1 glassfish nogroup 18892 Apr 24 2020
> ./appBaseNonWwwNumbeo/ROOT/ads.txt
> root@condor1796 /home/glassfish/apache-tomcat-8.5.5 # unzip -v
> appBaseNonWwwNumbeo/ROOT.war | grep "ads.txt"
>18892 Stored18892 0% 2020-04-24 16:13 dae45795 ads.txt
>
> Fine, all files are 18892 bytes.
>
> From server.xml:
> unpackWARs="true" autoDeploy="true">
> numbeo.com
>
> I did remove work, kill Catalina and Started it again.
>
> It's still serving old file! Checked with curl and browser (with delete
> cache and reload).
> apache-tomcat-8.5.5
>
> I'm totally confused. What is happening?
>
>
>
>
Tomcat serving old TXT file even after deleting work directory and rebooted
I've changed ads.txt did upload ROOT.war file and it now contains new version which is 18892 bytes old, checked files in the filesystem: root@condor1796 /home/glassfish/apache-tomcat-8.5.5 # find . | grep "ads.txt" | xargs ls -l -rw-r- 1 glassfish nogroup 18892 Apr 24 2020 ./appBaseLivingCost/ROOT/ads.txt -rw-r- 1 glassfish nogroup 18892 Apr 24 2020 ./appBaseNonWwwNumbeo/ROOT/ads.txt root@condor1796 /home/glassfish/apache-tomcat-8.5.5 # unzip -v appBaseNonWwwNumbeo/ROOT.war | grep "ads.txt" 18892 Stored18892 0% 2020-04-24 16:13 dae45795 ads.txt Fine, all files are 18892 bytes. >From server.xml: numbeo.com I did remove work, kill Catalina and Started it again. It's still serving old file! Checked with curl and browser (with delete cache and reload). apache-tomcat-8.5.5 I'm totally confused. What is happening?
Re: Large file uploads via https fail on Tomcat 9
Glad that helped!* Manuel Dominguez Sarmiento* On 24/04/2020 06:42, OIT Nua wrote: Thanks, again, for the assistance. Upgrading to 9.0.34 has resolved the issue.
RE: Large file uploads via https fail on Tomcat 9
Thanks, again, for the assistance. Upgrading to 9.0.34 has resolved the issue.
