Tomcat 8: Random 404 and 505 errors
Hi, We are seeing random 404 and 505 errors with GET and POST requests. Out of the 10 million daily requests in one cluster, there are 2-3 such 404 errors. In another cluster with around 100 million daily requests, we are seeing 20-30 404s on average per day. Requests with a 505 error are rare and only happen once or twice a week. The same requests were successful a few milliseconds before and after the error. The servlet that handles these requests always returns a 200 with an appropriate error response if it fails, but never a 404. So, it is surprising to see a 404 response. We've only had this issue in our test environment (similar to prod) a couple of times, but we can't reproduce it again. I've found a similar issue reported here: https://lists.apache.org/thread/rxypsggr36joq8fcd44khjky9w4qf4xx Deployment Details: Tomcat Version: 8.5.88. Built from source. Java Version: JDK 17. Network Topology: Direct server to server internal calls with no load balancers in between. Are there any known issues that could be causing this, and are there any specific logs we can enable that might help us debug this issue further ? -- Thanks Pavan
Re: CredentialHandler not working for MD5
Peter, On 11/16/23 13:06, Peter Otto wrote: 1. Configure BASIC auth with clear-text passwords in the Realm and get that working. 2. Switch to DIGEST auth with clear-text passwords in the Realm and get that working. 3. Then configure DIGEST auth and digested passwords in the Realm. Hi Chris, Step 1 & 2 work Good. Step 3 will not work with the clear txt password, only the digested password, which means the text password in tomcat-users.xml. In past versions of Tomcat, the clear text password would work. What does your Authentication request header look like? On line # 1154 in Realmbase.java we read. String digestValue = username + ":" + realmName + ":" + getPassword(username); The method getPassword(username) is using the digested password, when it should use the clear text password. Here is how I run digest in powershell. .\digest.bat -a MD5 -i 1 -s 0 tomcat:UserDatabase:nobueno RealmBase.java is not using the clear text password, instead it is using the digested password. This will return false for the manager access. When I replace the getPassword(username) and replace it with the clear text password, it will then WORK. How did you configure things for Mark's #3 task above? Including the commands you used to generate the stored-credential? -chris - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Java/Tomcat is being killed by the Linux OOM killer for using a huge amount of RAM. How can I know what was going on inside my app (& Tomcat & the JVM) to make that happen?
Brian, On 11/16/23 15:26, Brian Braun wrote: First of all, this is my stack: - Ubuntu 22.04.3 on x86/64 with 2GM of physical RAM that has been enough for years. - Java 11.0.20.1+1-post-Ubuntu-0ubuntu122.04 / openjdk 11.0.20.1 2023-08-24 - Tomcat 9.0.58 (JAVA_OPTS="-Djava.awt.headless=true -Xmx900m -Xms16m ..") Don't bother setting a 16M initial heap and a maximum of 900M. Just set them both to 900M. That will cause the JVM to request all of that heap up front and lessen the chances of a native OOME. There are certainly still plenty of reasons the process could use more heap than that, of course. - My app, which I developed myself, and has been running without any OOM crashes for years Well, a couple of weeks ago my website started crushing about every 5-7 days. Between crashes the RAM usage is fine and very steady (as it has been for years) and it uses just about 50% of the "Max memory" (according to what the Tomcat Manager server status shows). The 3 types of G1 heap are steady and low. And there are no leaks as far as I can tell. And I haven't made any significant changes to my app in the last months. I think your problem is native-heap and not Java-heap. What does 'top' say? You are looking for the "RES" (Resident Size) and "VIRT" (Virtual Size) numbers. That's what the process is REALLY using. How big is your physical RAM? What does this output while running your application (after fixing the heap at 900M)? $ free -m What else is running on the machine? Do you have swap enabled? When my website crashes, I can see on the Ubuntu log that some process has invoked the "oom-killer" and that this killer investigates which process is using most of the RAM and it is Tomcat/Java so it kills it. This is what I see on the log when it was Nginx that invoked the OOM-killer: Nov 15 15:23:54 ip-172-31-89-211 kernel: [366008.597771] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=nginx.service,mems_allowed=0,global_oom,task_memcg=/system.slice/tomcat9.service,task=java,pid=470,uid=998 Nov 15 15:23:54 ip-172-31-89-211 kernel: [366008.597932] Out of memory: Killed process 470 (java) total-vm:4553056kB, anon-rss:1527944kB, file-rss:2872kB, shmem-rss:0kB, UID:998 pgtables:3628kB oom_score_adj:0 I would like to be able to know what was happening inside the JVM when it was using too much RAM and deserved to be killed. Was it a problem in Java not associated with Tomcat or my app? Was it Tomcat itself that ate too much RAM? I doubt it. Was it my application? If it was my application (and I have to assume it was), how/why was it using all that RAM? What were the objects, threads, etc that were involved in the crash? What part of the heap memory was using all that RAM? Probably native heap. Java 11 is mature and there are likely no leaks in the JVM itself. If your code was using too much Java heap, you'd get OutOfMemoryErrors thrown in the JVM but not Linux oom-killer. But certain native libraries can leak. I seem to recall libgzip or something like that can leak if you aren't careful. My guess is that you are actually just running very very close to what your hardware can support. Do you actually need 900M of heap to run your application? We ran for years at $work with a 64M heap and only expended it when we started getting enough concurrent users to /have/ to expand the heap. This can happen at any time, like at 4am so I can not run to the computer to see what was going on at that moment. I need some way to get a detailed log of what was going on when the crush took place. So my question is, what tool should I use to investigate these crashes? I have started trying to make "New Relic" work since it seems that this service could help me, but I am having some problems making it work and I still don't know if this would be a solution in the first place. So, while I struggle with New Relic, I would appreciate your suggestions. You can get a lot of information by configuring your application to dump the heap on OOME, but you aren't getting an OOME so that's kind of off the table. I would enable GC logging for sure. That will tell you the status of the Java heap, but not the native memory spaces. But you may find that the process is performing a GC when it dies or you can see what was happening up to the point of the kill. Is there any pattern to when it crashes? For example... is 04:00 a popular time for it to die? Maybe you have a process that runs periodically that needs a lot of RAM temporarily. -chris - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Java/Tomcat is being killed by the Linux OOM killer for using a huge amount of RAM. How can I know what was going on inside my app (& Tomcat & the JVM) to make that happen?
Hello, First of all, this is my stack: - Ubuntu 22.04.3 on x86/64 with 2GM of physical RAM that has been enough for years. - Java 11.0.20.1+1-post-Ubuntu-0ubuntu122.04 / openjdk 11.0.20.1 2023-08-24 - Tomcat 9.0.58 (JAVA_OPTS="-Djava.awt.headless=true -Xmx900m -Xms16m ..") - My app, which I developed myself, and has been running without any OOM crashes for years Well, a couple of weeks ago my website started crushing about every 5-7 days. Between crashes the RAM usage is fine and very steady (as it has been for years) and it uses just about 50% of the "Max memory" (according to what the Tomcat Manager server status shows). The 3 types of G1 heap are steady and low. And there are no leaks as far as I can tell. And I haven't made any significant changes to my app in the last months. When my website crashes, I can see on the Ubuntu log that some process has invoked the "oom-killer" and that this killer investigates which process is using most of the RAM and it is Tomcat/Java so it kills it. This is what I see on the log when it was Nginx that invoked the OOM-killer: Nov 15 15:23:54 ip-172-31-89-211 kernel: [366008.597771] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=nginx.service,mems_allowed=0,global_oom,task_memcg=/system.slice/tomcat9.service,task=java,pid=470,uid=998 Nov 15 15:23:54 ip-172-31-89-211 kernel: [366008.597932] Out of memory: Killed process 470 (java) total-vm:4553056kB, anon-rss:1527944kB, file-rss:2872kB, shmem-rss:0kB, UID:998 pgtables:3628kB oom_score_adj:0 I would like to be able to know what was happening inside the JVM when it was using too much RAM and deserved to be killed. Was it a problem in Java not associated with Tomcat or my app? Was it Tomcat itself that ate too much RAM? I doubt it. Was it my application? If it was my application (and I have to assume it was), how/why was it using all that RAM? What were the objects, threads, etc that were involved in the crash? What part of the heap memory was using all that RAM? This can happen at any time, like at 4am so I can not run to the computer to see what was going on at that moment. I need some way to get a detailed log of what was going on when the crush took place. So my question is, what tool should I use to investigate these crashes? I have started trying to make "New Relic" work since it seems that this service could help me, but I am having some problems making it work and I still don't know if this would be a solution in the first place. So, while I struggle with New Relic, I would appreciate your suggestions. Thanks in advance!
Re: Wondering about tomcat-users.xml could not be found
Peter,
On 11/16/23 14:19, l...@kreuser.name wrote:
Hi Chris*,
Am 16.11.2023 um 20:12 schrieb Christopher Schultz
:
Christoph,
On 11/15/23 10:32, Christoph Kukulies wrote:
I'm running tomcat9 under Ubuntu 22.04 with an haproxy 2.8 in front of it.
I'm wondering about the following in the logs:
Nov 15 16:19:23 mail tomcat9[832]: Reloading memory user database
[UserDatabase] from updated source [file:/var/lib/tomcat9/conf/tomcat-users.xml]
Nov 15 16:19:23 mail tomcat9[832]: The specified user database
[conf/tomcat-users.xml] could not be found
Nov 15 16:19:33 mail tomcat9[832]: Reloading memory user database
[UserDatabase] from updated source [file:/var/lib/tomcat9/conf/tomcat-users.xml]
Nov 15 16:19:33 mail tomcat9[832]: The specified user database
[conf/tomcat-users.xml] could not be found
Nov 15 16:19:43 mail tomcat9[832]: Reloading memory user database
[UserDatabase] from updated source [file:/var/lib/tomcat9/conf/tomcat-users.xml]
Nov 15 16:19:43 mail tomcat9[832]: The specified user database
[conf/tomcat-users.xml] could not be found
Nov 15 16:19:53 mail tomcat9[832]: Reloading memory user database
[UserDatabase] from updated source [file:/var/lib/tomcat9/conf/tomcat-users.xml]
Nov 15 16:19:53 mail tomcat9[832]: The specified user database
[conf/tomcat-users.xml] could not be found
File /var/lib/tomcat9/conf/tomcat-users.xml is definitely there.
It occurs every 10 seconds.
Don't know who is causing this and why. Permissions? Ownership wrong?
-rw-r- 1 root root 2756 Jan 15 2022 tomcat-users.xml
Believe the ownership was wrong. Maybe it came from migrating an old
installation.
What are the correct perms/ownership in /var/lib/tomcat9 and below?
What is the user-owner of the JVM process?
Check that all of the above would be both readable and executable by that user:
ls -ld /var
ls -ld /var/lib
ls -ld /var/lib/tomcat9
ls -ld /var/lib/tomcat9/conf
... and of course that the JVM user can read
/var/lib/tomcat9/conf/tomcat-users.xml which I assume is true since you said
you already checked it.
What is the cwd of the JVM process?
The first message ("reloading") has the full path, and the second message ("file not
found") only mentions a relative path. I wonder if that is the difference.
Could it be that the second path relates to a missing env-Variable
$CATALINA_BASE or $CATALINA_HOME?
Unlikely. Tomcat always determines the values for catalina.home and
catalina.base before launching the JVM. After that, only those system
properties are consulted.
But it's possible there is some sloppy code somewhere that is using
cwd-relative paths.
-chris
-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
Re: AW: FileUpload class not working with Tomcat 10.1
Mark,
Apologies for not replying earlier; looks like you have made good
progress. See below.
On 11/14/23 12:19, Mark Foley wrote:
Anyway, enough griping! I have gotten it partially working thanks to your
suggested link, and particulary you suggestion to put the servlet info in
web.xml. I've put the following in WEB-INF/web.xml:
uploadfile
/schDistImportResultsX.jsp
/tmp
20848820
418018841
1048576
uploadfile
/schDistImportResultsX.jsp
I've only changed the and tags above. The others are
as monkey-typed from your link example. I'll research the other parameters
later.
My jsp code is now:
<%@ page import="javax.servlet.annotation.MultipartConfig.*" %>
Nope, not for Tomcat 10. You need to use the jakarta package names.
Besides, you don't need the MultipartConfig in your code, anyway.
You need /either/ an annotation (dicey in JSP code) /or/ an XML config,
so the XML should be sufficient. (But you should switch to a proper
servlet. DO IT! :)
if((contentType != null) && contentType.startsWith("multipart/form-data;"))
{
InputStream inp = null;
DataInputStream ins = null;
Part fileUpload = request.getPart("taxResults");
if(fileUpload != null)
{
inp = fileUpload.getInputStream();
ins = new DataInputStream(inp);
}
while ((inp != null) && (ins.available() != 0))
{
String transaction = ins.readLine();
out.println("" + transaction);
}
ins.close();
inp.close();
I would use try-with-resources like this:
try (InputStream in = , DataInputStream ins = ...) {
}
Since you have no try/catch, your code can leak file handles and stuff
like that. Yuck. With try-with-resources, you don't even need the calls
to InputStream.close.
This actually worked I will experiment with it more and may be back with
more questions (e.g. do I really need the web.xml? Could I not do:
"inp = fileUpload.getInputStream(mypath);"). But ... maybe later.
Vielen Dank!!! --Mark
Na klar
-chris
-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Partitioned cookies
Adam,
On 11/15/23 09:06, Adam Warfield wrote:
The Rfc6265CookieProcessor supports setting the SameSite cookie
attribute but starting in 2024, browsers will begin enforcing the
newer "Partitioned" attribute for third-party cookies.
Is there a way to set this attribute within Tomcat for things like the
JSESSIONID and XSRF-TOKEN cookies?
Wait... are you using cookies for CSRF tokens? That doesn't really
provide much protection. Your CSRF cookie will be transmitted along with
any request, even "forged" requests.
Are you responsible for the primary web application, here, or are you
responsible for a third-party site such as an advertiser, back-end
service, etc.?
This affects any webapps that are embedded within iframes across
domains where those cookies will be rejected if not partitioned.
If you migrate to Tomcat 10.1 or later (with Jakarta Servlet APIs),
there is Cookie.setAttributeString name, String value)[1]
If you cannot upgrade to Tomcat 10 in time, then you can simply resort
to setting the headers directly:
response.addHeader("Set-Cookie", "XSRF-TOKEN=foo; Partitioned");
-chris
[1]
https://jakarta.ee/specifications/servlet/6.0/apidocs/jakarta.servlet/jakarta/servlet/http/cookie#setAttribute(java.lang.String,java.lang.String)
-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Wondering about tomcat-users.xml could not be found
Hi Chris*,
> Am 16.11.2023 um 20:12 schrieb Christopher Schultz
> :
>
> Christoph,
>
> On 11/15/23 10:32, Christoph Kukulies wrote:
>> I'm running tomcat9 under Ubuntu 22.04 with an haproxy 2.8 in front of it.
>> I'm wondering about the following in the logs:
>> Nov 15 16:19:23 mail tomcat9[832]: Reloading memory user database
>> [UserDatabase] from updated source
>> [file:/var/lib/tomcat9/conf/tomcat-users.xml]
>> Nov 15 16:19:23 mail tomcat9[832]: The specified user database
>> [conf/tomcat-users.xml] could not be found
>> Nov 15 16:19:33 mail tomcat9[832]: Reloading memory user database
>> [UserDatabase] from updated source
>> [file:/var/lib/tomcat9/conf/tomcat-users.xml]
>> Nov 15 16:19:33 mail tomcat9[832]: The specified user database
>> [conf/tomcat-users.xml] could not be found
>> Nov 15 16:19:43 mail tomcat9[832]: Reloading memory user database
>> [UserDatabase] from updated source
>> [file:/var/lib/tomcat9/conf/tomcat-users.xml]
>> Nov 15 16:19:43 mail tomcat9[832]: The specified user database
>> [conf/tomcat-users.xml] could not be found
>> Nov 15 16:19:53 mail tomcat9[832]: Reloading memory user database
>> [UserDatabase] from updated source
>> [file:/var/lib/tomcat9/conf/tomcat-users.xml]
>> Nov 15 16:19:53 mail tomcat9[832]: The specified user database
>> [conf/tomcat-users.xml] could not be found
>> File /var/lib/tomcat9/conf/tomcat-users.xml is definitely there.
>> It occurs every 10 seconds.
>> Don't know who is causing this and why. Permissions? Ownership wrong?
>> -rw-r- 1 root root 2756 Jan 15 2022 tomcat-users.xml
>> Believe the ownership was wrong. Maybe it came from migrating an old
>> installation.
>> What are the correct perms/ownership in /var/lib/tomcat9 and below?
>
> What is the user-owner of the JVM process?
>
> Check that all of the above would be both readable and executable by that
> user:
>
> ls -ld /var
> ls -ld /var/lib
> ls -ld /var/lib/tomcat9
> ls -ld /var/lib/tomcat9/conf
>
> ... and of course that the JVM user can read
> /var/lib/tomcat9/conf/tomcat-users.xml which I assume is true since you said
> you already checked it.
>
> What is the cwd of the JVM process?
>
> The first message ("reloading") has the full path, and the second message
> ("file not found") only mentions a relative path. I wonder if that is the
> difference.
>
Could it be that the second path relates to a missing env-Variable
$CATALINA_BASE or $CATALINA_HOME?
Peter
> -chris
>
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>
-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Wondering about tomcat-users.xml could not be found
Christoph,
On 11/15/23 10:32, Christoph Kukulies wrote:
I'm running tomcat9 under Ubuntu 22.04 with an haproxy 2.8 in front of it.
I'm wondering about the following in the logs:
Nov 15 16:19:23 mail tomcat9[832]: Reloading memory user database
[UserDatabase] from updated source
[file:/var/lib/tomcat9/conf/tomcat-users.xml]
Nov 15 16:19:23 mail tomcat9[832]: The specified user database
[conf/tomcat-users.xml] could not be found
Nov 15 16:19:33 mail tomcat9[832]: Reloading memory user database
[UserDatabase] from updated source
[file:/var/lib/tomcat9/conf/tomcat-users.xml]
Nov 15 16:19:33 mail tomcat9[832]: The specified user database
[conf/tomcat-users.xml] could not be found
Nov 15 16:19:43 mail tomcat9[832]: Reloading memory user database
[UserDatabase] from updated source
[file:/var/lib/tomcat9/conf/tomcat-users.xml]
Nov 15 16:19:43 mail tomcat9[832]: The specified user database
[conf/tomcat-users.xml] could not be found
Nov 15 16:19:53 mail tomcat9[832]: Reloading memory user database
[UserDatabase] from updated source
[file:/var/lib/tomcat9/conf/tomcat-users.xml]
Nov 15 16:19:53 mail tomcat9[832]: The specified user database
[conf/tomcat-users.xml] could not be found
File /var/lib/tomcat9/conf/tomcat-users.xml is definitely there.
It occurs every 10 seconds.
Don't know who is causing this and why. Permissions? Ownership wrong?
-rw-r- 1 root root 2756 Jan 15 2022 tomcat-users.xml
Believe the ownership was wrong. Maybe it came from migrating an old
installation.
What are the correct perms/ownership in /var/lib/tomcat9 and below?
What is the user-owner of the JVM process?
Check that all of the above would be both readable and executable by
that user:
ls -ld /var
ls -ld /var/lib
ls -ld /var/lib/tomcat9
ls -ld /var/lib/tomcat9/conf
... and of course that the JVM user can read
/var/lib/tomcat9/conf/tomcat-users.xml which I assume is true since you
said you already checked it.
What is the cwd of the JVM process?
The first message ("reloading") has the full path, and the second
message ("file not found") only mentions a relative path. I wonder if
that is the difference.
-chris
-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
Re: CredentialHandler not working for MD5
1. Configure BASIC auth with clear-text passwords in the Realm and get that working. 2. Switch to DIGEST auth with clear-text passwords in the Realm and get that working. 3. Then configure DIGEST auth and digested passwords in the Realm. Hi Chris, Step 1 & 2 work Step 3 will not work with the clear txt password, only the digested password, which means the text password in tomcat-users.xml. In past versions of Tomcat, the clear text password would work. On line # 1154 in Realmbase.java we read. String digestValue = username + ":" + realmName + ":" + getPassword(username); The method getPassword(username) is using the digested password, when it should use the clear text password. Here is how I run digest in powershell. .\digest.bat -a MD5 -i 1 -s 0 tomcat:UserDatabase:nobueno RealmBase.java is not using the clear text password, instead it is using the digested password. This will return false for the manager access. When I replace the getPassword(username) and replace it with the clear text password, it will then WORK. This e-mail and any files transmitted with it are the property of Arthrex, Inc. and/or its affiliates, are confidential, and are intended solely for the use of the individual or entity to whom this e-mail is addressed. If you are not one of the named recipient(s) or otherwise have reason to believe that you have received this message in error, please notify the sender at 239-643-5553 and delete this message immediately from your computer. Any other use, retention, dissemination forwarding, printing or copying of this e-mail is strictly prohibited. Please note that any views or opinions presented in this email are solely those of the author and do not necessarily represent those of the company. Finally, while Arthrex uses virus protection, the recipient should check this email and any attachments for the presence of viruses. The company accepts no liability for any damage caused by any virus transmitted by this email.
