Re: Working dir incorrect tomcat 9.10?
Ok, thanks for the clarification and of course there are multiple ways to solve this; I just ran in to problems since the behaviour seemed to have changed between the versions but I've upgrade the whole stack from OS to java to tomcat so its was hard to isolate the change and I made the wrong assumption that the CWD was set to CATALINA_BASE. With your information I got to the root of why the CWD has changed to '/' and it was a simple as I had not set a home dir for the tomcat9 user and thus the user-dir was automatically set to '/' when the systemd-script started tomcat as the tomcat9 user.. Br, Kalle - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Working dir incorrect tomcat 9.10?
Hi!
I'm in the process of upgrading from tomcat 8 to 9 and was running
into a probelm with velocity not beeing able to create the default log
file, ./velocity.log and after some troubleshooting it seems it is
trying to create it in the root of the file system instead of the
current working directory, which i thought would be CATALINA_BASE.
I just did some test and logging in the webapp:
log.info("Working path: " + new File(".").getAbsolutePath());
is reporting "Working path: /.
Also, I could do a workaround by creating /velocity.log manually and
give the tomcat9 user ownership which strengthen my belief that there
is something strange with the working path..
>From the logs it seems that tomcat is picking up the correct settings,
see below, so I really cant get my head around it..
This has been workign without a problem from tomcat6 to tomcat8, is
there some configuration change I may have missed?
28-Jun-2018 08:42:28.896 INFO [main]
org.apache.catalina.startup.VersionLoggerListener.log Server version:
Apache Tomcat/9.0.10
28-Jun-2018 08:42:28.897 INFO [main]
org.apache.catalina.startup.VersionLoggerListener.log Server built:
Jun 20 2018 17:32:21 UTC
28-Jun-2018 08:42:28.898 INFO [main]
org.apache.catalina.startup.VersionLoggerListener.log Server number:
9.0.10.0
28-Jun-2018 08:42:28.899 INFO [main]
org.apache.catalina.startup.VersionLoggerListener.log OS Name:
Linux
28-Jun-2018 08:42:28.899 INFO [main]
org.apache.catalina.startup.VersionLoggerListener.log OS Version:
4.15.0-23-generic
28-Jun-2018 08:42:28.900 INFO [main]
org.apache.catalina.startup.VersionLoggerListener.log Architecture:
amd64
28-Jun-2018 08:42:28.900 INFO [main]
org.apache.catalina.startup.VersionLoggerListener.log Java Home:
/usr/lib/jvm/java-11-openjdk-amd64
28-Jun-2018 08:42:28.901 INFO [main]
org.apache.catalina.startup.VersionLoggerListener.log JVM Version:
10.0.1+10-Ubuntu-3ubuntu1
28-Jun-2018 08:42:28.902 INFO [main]
org.apache.catalina.startup.VersionLoggerListener.log JVM Vendor:
Oracle Corporation
28-Jun-2018 08:42:28.902 INFO [main]
org.apache.catalina.startup.VersionLoggerListener.log CATALINA_BASE:
/ssd/opt/apache-tomcat-9.0.10
28-Jun-2018 08:42:28.903 INFO [main]
org.apache.catalina.startup.VersionLoggerListener.log CATALINA_HOME:
/ssd/opt/apache-tomcat-9.0.10
28-Jun-2018 08:42:28.904 INFO [main]
org.apache.catalina.startup.VersionLoggerListener.log Command line
argument: --add-opens=java.base/java.lang=ALL-UNNAMED
28-Jun-2018 08:42:28.905 INFO [main]
org.apache.catalina.startup.VersionLoggerListener.log Command line
argument: --add-opens=java.base/java.io=ALL-UNNAMED
28-Jun-2018 08:42:28.905 INFO [main]
org.apache.catalina.startup.VersionLoggerListener.log Command line
argument: --add-opens=java.rmi/sun.rmi.transport=ALL-UNNAMED
28-Jun-2018 08:42:28.906 INFO [main]
org.apache.catalina.startup.VersionLoggerListener.log Command line
argument: -Djava.util.logging.config.file=/opt/tomcat/conf/logging.pro
perties
28-Jun-2018 08:42:28.906 INFO [main]
org.apache.catalina.startup.VersionLoggerListener.log Command line
argument: -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogMa
nager
28-Jun-2018 08:42:28.907 INFO [main]
org.apache.catalina.startup.VersionLoggerListener.log Command line
argument: -Dfile.encoding=ISO-8859-1
28-Jun-2018 08:42:28.907 INFO [main]
org.apache.catalina.startup.VersionLoggerListener.log Command line
argument: -Dnet.sf.ehcache.skipUpdateCheck=true
28-Jun-2018 08:42:28.908 INFO [main]
org.apache.catalina.startup.VersionLoggerListener.log Command line
argument: -XX:+CMSClassUnloadingEnabled
28-Jun-2018 08:42:28.908 INFO [main]
org.apache.catalina.startup.VersionLoggerListener.log Command line
argument: -Djdk.tls.ephemeralDHKeySize=2048
28-Jun-2018 08:42:28.909 INFO [main]
org.apache.catalina.startup.VersionLoggerListener.log Command line
argument: -Djava.protocol.handler.pkgs=org.apache.catalina.webresource
s
28-Jun-2018 08:42:28.909 INFO [main]
org.apache.catalina.startup.VersionLoggerListener.log Command line
argument: -Dorg.apache.catalina.security.SecurityListener.UMASK=0027
28-Jun-2018 08:42:28.910 INFO [main]
org.apache.catalina.startup.VersionLoggerListener.log Command line
argument: -Xms512m
28-Jun-2018 08:42:28.910 INFO [main]
org.apache.catalina.startup.VersionLoggerListener.log Command line
argument: -Xmx1G
28-Jun-2018 08:42:28.912 INFO [main]
org.apache.catalina.startup.VersionLoggerListener.log Command line
argument: --add-modules=java.xml.bind
28-Jun-2018 08:42:28.913 INFO [main]
org.apache.catalina.startup.VersionLoggerListener.log Command line
argument: --add-modules=java.xml.ws
28-Jun-2018 08:42:28.914 INFO [main]
org.apache.catalina.startup.VersionLoggerListener.log Command line
argument: -Dignore.endorsed.dirs=
28-Jun-2018 08:42:28.915 INFO [main]
org.apache.catalina.startup.VersionLoggerListener.log Command line
argument: -Dcatalina.base=/opt/tomcat
28-Jun-2018 08:42:28.918 INFO [main]
Working dir incorrect tomcat 9.10?
Hi!
I'm in the process of upgrading from tomcat 8 to 9 and was running into a
probelm with velocity not beeing able to create the default log file,
./velocity.log and after some troubleshooting it seems it is trying to
create it in the root of the file system instead of the current working
directory, which i thought would be CATALINA_BASE.
I just did some test and logging in the webapp:
log.info("Working path: " + new File(".").getAbsolutePath());
would report "Working path: /.
And I could do a workaround by creating /velocity.log manually and give the
tomcat9 user ownership.
>From the logs it seems that tomcat is picking up the correct settings, see
below, so I really cant get my head around it..
This has been workign without a problem from tomcat6 to tomcat8, is there
some configuration change I may have missed?
28-Jun-2018 08:42:28.896 INFO [main]
org.apache.catalina.startup.VersionLoggerListener.log Server
version:Apache Tomcat/9.0.10
28-Jun-2018 08:42:28.897 INFO [main]
org.apache.catalina.startup.VersionLoggerListener.log Server
built: Jun 20 2018 17:32:21 UTC
28-Jun-2018 08:42:28.898 INFO [main]
org.apache.catalina.startup.VersionLoggerListener.log Server
number: 9.0.10.0
28-Jun-2018 08:42:28.899 INFO [main]
org.apache.catalina.startup.VersionLoggerListener.log OS
Name: Linux
28-Jun-2018 08:42:28.899 INFO [main]
org.apache.catalina.startup.VersionLoggerListener.log OS
Version:4.15.0-23-generic
28-Jun-2018 08:42:28.900 INFO [main]
org.apache.catalina.startup.VersionLoggerListener.log
Architecture: amd64
28-Jun-2018 08:42:28.900 INFO [main]
org.apache.catalina.startup.VersionLoggerListener.log Java
Home: /usr/lib/jvm/java-11-openjdk-amd64
28-Jun-2018 08:42:28.901 INFO [main]
org.apache.catalina.startup.VersionLoggerListener.log JVM
Version: 10.0.1+10-Ubuntu-3ubuntu1
28-Jun-2018 08:42:28.902 INFO [main]
org.apache.catalina.startup.VersionLoggerListener.log JVM
Vendor:Oracle Corporation
28-Jun-2018 08:42:28.902 INFO [main]
org.apache.catalina.startup.VersionLoggerListener.log
CATALINA_BASE: /ssd/opt/apache-tomcat-9.0.10
28-Jun-2018 08:42:28.903 INFO [main]
org.apache.catalina.startup.VersionLoggerListener.log
CATALINA_HOME: /ssd/opt/apache-tomcat-9.0.10
28-Jun-2018 08:42:28.904 INFO [main]
org.apache.catalina.startup.VersionLoggerListener.log Command line
argument: --add-opens=java.base/java.lang=ALL-UNNAMED
28-Jun-2018 08:42:28.905 INFO [main]
org.apache.catalina.startup.VersionLoggerListener.log Command line
argument: --add-opens=java.base/java.io=ALL-UNNAMED
28-Jun-2018 08:42:28.905 INFO [main]
org.apache.catalina.startup.VersionLoggerListener.log Command line
argument: --add-opens=java.rmi/sun.rmi.transport=ALL-UNNAMED
28-Jun-2018 08:42:28.906 INFO [main]
org.apache.catalina.startup.VersionLoggerListener.log Command line
argument: -Djava.util.logging.config.file=/opt/tomcat/conf/logging.pro
perties
28-Jun-2018 08:42:28.906 INFO [main]
org.apache.catalina.startup.VersionLoggerListener.log Command line
argument: -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogMa
nager
28-Jun-2018 08:42:28.907 INFO [main]
org.apache.catalina.startup.VersionLoggerListener.log Command line
argument: -Dfile.encoding=ISO-8859-1
28-Jun-2018 08:42:28.907 INFO [main]
org.apache.catalina.startup.VersionLoggerListener.log Command line
argument: -Dnet.sf.ehcache.skipUpdateCheck=true
28-Jun-2018 08:42:28.908 INFO [main]
org.apache.catalina.startup.VersionLoggerListener.log Command line
argument: -XX:+CMSClassUnloadingEnabled
28-Jun-2018 08:42:28.908 INFO [main]
org.apache.catalina.startup.VersionLoggerListener.log Command line
argument: -Djdk.tls.ephemeralDHKeySize=2048
28-Jun-2018 08:42:28.909 INFO [main]
org.apache.catalina.startup.VersionLoggerListener.log Command line
argument: -Djava.protocol.handler.pkgs=org.apache.catalina.webresource
s
28-Jun-2018 08:42:28.909 INFO [main]
org.apache.catalina.startup.VersionLoggerListener.log Command line
argument: -Dorg.apache.catalina.security.SecurityListener.UMASK=0027
28-Jun-2018 08:42:28.910 INFO [main]
org.apache.catalina.startup.VersionLoggerListener.log Command line
argument: -Xms512m
28-Jun-2018 08:42:28.910 INFO [main]
org.apache.catalina.startup.VersionLoggerListener.log Command line
argument: -Xmx1G
28-Jun-2018 08:42:28.912 INFO [main]
org.apache.catalina.startup.VersionLoggerListener.log Command line
argument: --add-modules=java.xml.bind
28-Jun-2018 08:42:28.913 INFO [main]
org.apache.catalina.startup.VersionLoggerListener.log Command line
argument: --add-modules=java.xml.ws
28-Jun-2018 08:42:28.914 INFO [main]
org.apache.catalina.startup.VersionLoggerListener.log Command line
argument: -Dignore.endorsed.dirs=
28-Jun-2018 08:42:28.915 INFO [main]
org.apache.catalina.startup.VersionLoggerListener.log Command line
argument: -Dcatalina.base=/opt/tomcat
28-Jun-2018 08:42:28.918 INFO [main]
org.apache.catalina.startup.VersionLoggerListener.log Command
REST issue
Some foundation facts:
Development environment using NetBeans 8.0.2, Tomcat 8.0.32, Java
jdk1.8.0_121 and Postman.
The code:
I have a simple hello world servlet for testing (in production, the
service will serve as an endpoint for information from another
organization):
package com.tsr.webservices;
import javax.jws.WebService;
import javax.jws.WebMethod;
import javax.jws.WebParam;
import javax.ws.rs.Consumes;
import javax.ws.rs.GET;
import javax.ws.rs.POST;
import javax.ws.rs.Path;
import javax.ws.rs.PathParam;
import javax.ws.rs.Produces;
import javax.ws.rs.core.MediaType;
import org.json.JSONObject;
/**
*
* @author Carl
*/
@WebService(serviceName = "helloWorld")
@Path("/helloWorld")
public class AnoviaWebService {
/**
* Retrieves representation of an instance of helloWorld.HelloWorld
* @return an instance of java.lang.String
*/
@GET
@Produces("text/html")
public String getHtml() {
return "Hello, World!!";
}
}
The web.xml contains these fragments:
...
AnoviaWebService
org.glassfish.jersey.servlet.ServletContainer
jersey.config.server.provider.packages
com.tsr.webservice
1
...
AnoviaWebService
/helloWorld/*
...
Tomcat starts properly with no errors. Using Postman with either
http://192.168.0.117:8080/prod2test/helloWorld/getHtml
or
http://192.168.0.117:8080/prod2test/helloWorld
the first request after a rebuild/redeploy (we use a request filter and
I can see the request hitting the filter) returns a 200 OK but the jsp
that is displayed is the jsp that is displayed when the request failed
to pass the filter requirements. Subsequent requests return a 404, not
found error and I can see the request never hits the filter after that
first request.
Additional notes.
1. I am attempting to add this service to an existing application that
is quite large because, at some time in the not too distant future, the
entire application will be changed to run as a sevice so the client and
server sides can be decoupled.
2. I have successfully created/deployed/run a REST service using the
NetBeans templates but it has no web.xml and uses the
'ApplicationConfig' process to expose the service.
If anyone has any ideas, I would certainly appreciate them.
Thanks,
Carl
-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Vulnerability from PCI scan
Chris, On 11/2/2016 11:05 AM, Christopher Schultz wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Carl, On 11/1/16 6:05 PM, Carl K. wrote: On 11/1/2016 5:25 PM, Christopher Schultz wrote: Carl, On 11/1/16 5:11 PM, Carl K. wrote: Control Scan has returned this as a vulnerability in Tomcat 8.0.38: Vulnerable version of Apache Tomcat: 8.0.38 Risk: High (3) Port: 443/tcp Protocol: tcp Threat ID: web_dev_tomcatver Details: 404 Error Page Cross Site Scripting Vulnerability 12/21/09 Apache Tomcat is prone to a cross-site scripting vulnerability because it fails to properly sanitize user-supplied input. An attacker may leverage this issue to execute arbitrary script code in the browser of an unsuspecting user in the context of the affected site. Apache Tomcat mitigates HTTP_PROXY environment variable "httpoxy" issue I have read everything I can find and it still doesn't make sense... can someone help to point me in the correct direction? I am further puzzled because this is the first time this has come up and we run Tomcat for years... note that the date is listed as 12-21-2009. Technically, this is not a vulnerability in Tomcat (or any reverse-proxy, such as httpd) but it does represent a failure to protect stupid command-line utilities from making bad decisions about trusting environment variables. Long story short, if using the CGI Servlet, any headers coming from the request are set as HTTP_* environment variables on a script that is executed as a CGI script. Notably, python, Perl, and PHP (and others) use an environment variable called HTTP_PROXY to indicate the presence of a forward-proxy to be used for outgoing HTTP connections. Thus, setting a "Proxy" header in an HTTP request to Tomcat will result in a CGI script seeing that value in the HTTP_PROXY environment variable. This could present a problem in your environment, but is possible to mitigate in a number of different ways. https://www.apache.org/security/asf-httpoxy-response.txt I have no idea where your scanner got the date 2009-12-21. Perhaps they took the recently-disclosed CVE (CVE-2016-5388 -- note the year on that CVE identifier) and made a best-guess of when the product was first vulnerable. The first beta version of Tomcat 7 wasn't available until 2010, so perhaps they were considering Tomcat 6 as well. But Tomcat 6's history goes back well before that. Honestly, I think they may have picked that date out of the air. At any rate, you are safe if any of the following are true: 1. You don't use the CGI servlet 2. You don't use any scripts that use HTTP_PROXY in this manner (this is a weak criteria, since you may not KNOW if you are using such scripts) 3. You don't allow outgoing HTTP requests from your application servers, and no error messages produces by those scripts would leak any information like URLs, etc. 4. If you have a reverse-proxy (e.g. httpd) and explicitly remove any "proxy" headers from incoming HTTP requests Mitigation is possible through a variety of means. If you aren't vulnerable, this scan is likely to complain merely because of the version number of Tomcat and the fact that this CVE hasn't officially been closed, yet. That is about where I had gotten to. I really appreciate your quick and thorough response. I dug a bit, and it seems that this was fixed in Tomcat 8.0.38, as per the changelog[1]. Search for CVE-2016-5388 (or httpoxy) and you'll see it's in there. So your scanning is either incorrect or you have explicitly whitelisted the "PROXY" header for your CGIs. I suspect you ave no CGIs and the scanner is just dumping a list of things that *might* be vulnerabilities. - -chris That is what I am thinking and I have relayed that to ControlScan. Thanks,, Carl -BEGIN PGP SIGNATURE- Comment: GPGTools - http://gpgtools.org Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQIcBAEBCAAGBQJYGgDUAAoJEBzwKT+lPKRYzwwP/A5qW+36B0gYtse5r4QBjVi/ y/aixHjpG5RaEH9SlJSUethhkqr2HKmoDFRyXe2z6HQcK/9gFIBMYeButF2FrqIH 8M5AnkgzbepaFPKL8ZK63J2I898lvusmdqcDVpZ5ttJDPEZyj/quaRWunhyYYXb3 A4gwqsp6QuASf8x53/CiHfLKgE7r1oJsfZVcSKPRhUi2EFG26FJhuCdZAufH9mw7 SnHqdWxWBk34w4oC6eyrb6Z7fpK8XFgu3NvQ9cJbrX9ivJ3nqiwAKMyg0Q8l2Xda z5ERLCbKRw7GuxXeGrWHzIFfnreKGBVM3IuHwhjAtkg7bzWkTNPUr1ZrewBpDHjY xRJt1ahqK04H6ctTTsPd/Xhti4q4mGI8tjas8Gt38VicJEjPKc8EBOW94NF+kz64 jhp1KHcC7yIwbUHYAfm1IYRrQxo4PkDtZbDUdbum0gCYAFgvuLoVFib35fL0rtyu bfFNrI+taDjA57z4OqMx5/FvYj+gJhm41dR5SlC5ULkuOGz7WrNWXrYZztwsQs7n /kagYzIleuZZ7bitZIfjCxrGlQtPcOahAHWXItaxpUrae4VBXNWthouSfrV+mXSy WDOWCnnYzOc9OemgOxjeifPP9RlEFYow4r4go2kZg4Kxo9ihIT4uzC+YbEJu1oP9 sPGi1+GcIBhvYMrzOA8C =+w7Z -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsu
Re: Vulnerability from PCI scan
On 11/1/2016 5:25 PM, Christopher Schultz wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Carl, On 11/1/16 5:11 PM, Carl K. wrote: Control Scan has returned this as a vulnerability in Tomcat 8.0.38: Vulnerable version of Apache Tomcat: 8.0.38 Risk: High (3) Port: 443/tcp Protocol: tcp Threat ID: web_dev_tomcatver Details: 404 Error Page Cross Site Scripting Vulnerability 12/21/09 Apache Tomcat is prone to a cross-site scripting vulnerability because it fails to properly sanitize user-supplied input. An attacker may leverage this issue to execute arbitrary script code in the browser of an unsuspecting user in the context of the affected site. Apache Tomcat mitigates HTTP_PROXY environment variable "httpoxy" issue I have read everything I can find and it still doesn't make sense... can someone help to point me in the correct direction? I am further puzzled because this is the first time this has come up and we run Tomcat for years... note that the date is listed as 12-21-2009. Technically, this is not a vulnerability in Tomcat (or any reverse-proxy, such as httpd) but it does represent a failure to protect stupid command-line utilities from making bad decisions about trusting environment variables. Long story short, if using the CGI Servlet, any headers coming from the request are set as HTTP_* environment variables on a script that is executed as a CGI script. Notably, python, Perl, and PHP (and others) use an environment variable called HTTP_PROXY to indicate the presence of a forward-proxy to be used for outgoing HTTP connections. Thus, setting a "Proxy" header in an HTTP request to Tomcat will result in a CGI script seeing that value in the HTTP_PROXY environment variable. This could present a problem in your environment, but is possible to mitigate in a number of different ways. https://www.apache.org/security/asf-httpoxy-response.txt I have no idea where your scanner got the date 2009-12-21. Perhaps they took the recently-disclosed CVE (CVE-2016-5388 -- note the year on that CVE identifier) and made a best-guess of when the product was first vulnerable. The first beta version of Tomcat 7 wasn't available until 2010, so perhaps they were considering Tomcat 6 as well. But Tomcat 6's history goes back well before that. Honestly, I think they may have picked that date out of the air. At any rate, you are safe if any of the following are true: 1. You don't use the CGI servlet 2. You don't use any scripts that use HTTP_PROXY in this manner (this is a weak criteria, since you may not KNOW if you are using such scripts) 3. You don't allow outgoing HTTP requests from your application servers, and no error messages produces by those scripts would leak any information like URLs, etc. 4. If you have a reverse-proxy (e.g. httpd) and explicitly remove any "proxy" headers from incoming HTTP requests Mitigation is possible through a variety of means. If you aren't vulnerable, this scan is likely to complain merely because of the version number of Tomcat and the fact that this CVE hasn't officially been closed, yet. That is about where I had gotten to. I really appreciate your quick and thorough response. Carl - -chris -BEGIN PGP SIGNATURE- Comment: GPGTools - http://gpgtools.org Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQIcBAEBCAAGBQJYGQhlAAoJEBzwKT+lPKRYaxQP+waQq2j3Jb8L6VgFoBoMGvZl SVekwwOHR6L4aEhoLw/ETD1v7U7w0uapNmDgZ56ZtX83dr9S2OqWdtYF1xrVwM8m /Plwn8eR5oX+0Xq+FzSUS3OQp/I+kKdlU/4JhLtGGPgx2c7uW+mE8mCcfDYIn1ro fYMxvSmWkRLAJ8lRuIwQ4mA7UvkrTb2mlIQfceoKc/SbmVFFCNd49GIHD9uRi+GZ 9tPADCY82LVpDJxA3PmuT5skazuYUGuYnZ1j3FEUrUMnFlKjrDzxjJXiGz7UPfLa 8xmyP3DgaJTSJN6izwC0sNgdsqKiyM1A2/CPbLQtkLjaGB0WDzDq1rENRhHS9512 qg4EL4EPyfuwf+9k8v4Svm/quHTHD9HVs8cWrWEfJc3f1QWSSbLhOv1UcIE6wfNZ EBaBbJHZqIxG3ECvM+CgHjSIA1sQCObC5ltOIaQuLHAbUhvj1uDxcjj823sS5tZp LZuZOJNLo4NHWHpDu71YkzPbiQCuka7m6WkXgWDzLvyK2BxCXCJD/ifabbVh82+h PDXXuLYVSgYtQoQipcZ8qNlaMBik+IGpSzkp7YsEiHD38eJxxQEfjov4/MJtHjKs zfTwhjAT0GwpFicpHUXyzUNq3lDERQO6wlq3VRtgNxaKxFMqX9gn0AUYyMZ0NH2N dtrZR8YGCgO1QBGpCOcI =neSa -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Vulnerability from PCI scan
Control Scan has returned this as a vulnerability in Tomcat 8.0.38: Vulnerable version of Apache Tomcat: 8.0.38 Risk: High (3) Port: 443/tcp Protocol: tcp Threat ID: web_dev_tomcatver Details: 404 Error Page Cross Site Scripting Vulnerability 12/21/09 Apache Tomcat is prone to a cross-site scripting vulnerability because it fails to properly sanitize user-supplied input. An attacker may leverage this issue to execute arbitrary script code in the browser of an unsuspecting user in the context of the affected site. Apache Tomcat mitigates HTTP_PROXY environment variable "httpoxy" issue I have read everything I can find and it still doesn't make sense... can someone help to point me in the correct direction? I am further puzzled because this is the first time this has come up and we run Tomcat for years... note that the date is listed as 12-21-2009. Thanks, Carl
Re: threads vs. servlets
If I write a servlet such as the above, is there ever only once instance of it running? Don't confuse objects with threads. There is one instance of a particular servlet, but many threads may be executing in it concurrently, with each thread processing a separate request. I understand that each request is handled by a separate thread. But does each thread have its own copy of the servlet code? Or does each thread request the use of the servlet, wait until it is available, use it, and then release it back to be used by the next thread, sort of like a database connection? I'm pretty sure it is the former, but just wanted to check. I'd like to offer a suggestion: In multiple places, the FAQs about using this list have comments such as ...be sure to check the archives before asking a question... but don't have any links (or instructions) on HOW to do that! There's no point in repeating something in a myriad of places that you must have already read in order to sign up for the mailing list. As clearly stated on the mailing lists page (http://tomcat.apache.org/lists.html): Formatted archives are available in several places including the Apache Mail Archives, MARC, Nabble, and MarkMail. The raw mbox files are also available. That presumes that someone searching for an answer is a member of this list. I suspect that there are many, many more people who have download and are trying Tomcat than are here. It is very likely someone finds a reference to a discussion through Google, and thus don't come through the Apache page front door. I actually did go to the Apache page you referenced when I started searching, and saw that line. The Apache Mail Archives link takes you to non-searchable records of every email. The MARC link returns a ridiculous list of hundreds of additional links, one of which is tomcat users, which returns even more links, etc. I don't know WHAT Nabble is suppose to do...it seems to be about starting blogs. The MarkMail link is OK once you figure out what it does, but it is not as good as the simple link I provided. Oh, and none of the FAQ pages provide links or instructions on how to search the archive. So, no, you don't have to repeat the instructions in myriad places. Just simply, cleanly explain it (with examples) in one place and then include links in myriad places. It is like putting a contact us link on every page of a website instead of just the home page: it simply makes it easier for the user. (Some people call it user-friendly. Personally, I just call it being helpful.) - Carl Dreher - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
restricting access to images
I need to restrict access to a website's images, to people that have logged on, have authorization etc. I've searched though the Tomcat user's mailing list archives and didn't find a discussion that addressed this, so I thought I'd asked for some architectural guidance. My initial thought is to have the src parameter in an html img src=url / point to a servlet instead of a static image in the web app. The servlet would check the session and verify that the requester is logged-in and then return the appropriate image. Seems straight forward. Is there a better way? I read some threads about Tomcat filters but that seems like overkill. A related question is more fundamental: If I write a servlet such as the above, is there ever only once instance of it running? In other words, if I have 10 users hitting the site at once, does Tomcat create an instance of the servlet for each user so they all operate in parallel, or does queue-up the requests and send them to a single instance? By the way, if anyone here is administering this mailing list, I'd like to offer a suggestion: In multiple places, the FAQs about using this list have comments such as ...be sure to check the archives before asking a question... but don't have any links (or instructions) on HOW to do that! I had to resort to Google to find the archive, and then it took more time to find the *searchable* archive, which is entirely different. . A simple link to http://www.mail-archive.com/users@tomcat.apache.org/maillist.html; on those FAQ pages, as well as to the bottom of this list, would be very, very helpful. - Carl Dreher - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Consulting opportunity
We have what we believe is approximately a one week assignment to cluster several instances of Tomcat. The clustering should be relatively straight forward except that most of the classes on the server side extend a class which contains the datasource and is therefore not serializable. Although we believe we could do this ourselves from a technical point of view, we don't have the resources to spare. We are currently running Tomcat 8.0.9 against Java 8.0.20 running on the latest Slackware servers. We are located in Charleston, SC. If you have some interest, please email me directly. Thanks, Carl - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat 8.0.9 native library not found
On Sat, Aug 16, 2014 at 9:26 PM, Neil Aggarwal n...@jammconsulting.com wrote: [truncated] cd tomcat-native-1.1.30-src/jni/native ./configure --with-apr=/usr/bin/apr-1-config When I look in /usr/local/apr/lib, I see these files: libtcnative-1.a libtcnative-1.la libtcnative-1.so libtcnative-1.so.0 --- first, don't use tomcat-native-1.1.30, use tomcat-native-1.1.31 from http://tomcat.apache.org/download-native.cgi Including thee old version is just Apache Foundation sense of humor. then you say ./configure --with-apr=/usr/bin/apr-1-config Is APR really installed, and really there? Install APR first , maybe in /usr/local/apr/ Then intall the tomcat-native --with-apr=/usr/local/apr and maybe it will work
Re: Tomcat 8.0.9 native library not found
I can do /usr/bin/apr-1-config and I get the usage page for APR. I assume that means it is there. Don't assume. Your command lines said is was in /usr/lib/apr Is it in /usr/lib/apr? from what you said above, it isn't. I suggest you get APR from http://apr.apache.org/ and reinstall it. When I did, it went in /usr/lib/apr/ which is convenient
Tomcat cross-site scripting vulnerability
Our latest PCI scan using the Saint scanner shows the following: 404 Error Page Cross Site Scripting Vulnerability 12/21/09 Apache Tomcat is prone to a cross-site scripting vulnerability because it fails to properly sanitize user-supplied input. An attacker may leverage this issue to execute arbitrary script code in the browser of an unsuspecting user in the context of the affected site. Is there any way to mitigate this vulnerability (I suspect anyone using Tomcat is going to see the same thing)? Thanks, Carl
Re: Tomcat cross-site scripting vulnerability
On 7/4/2014 9:31 AM, Mark Thomas wrote: On 04/07/2014 14:12, carl wrote: Our latest PCI scan using the Saint scanner shows the following: 404 Error Page Cross Site Scripting Vulnerability 12/21/09 Apache Tomcat is prone to a cross-site scripting vulnerability because it fails to properly sanitize user-supplied input. An attacker may leverage this issue to execute arbitrary script code in the browser of an unsuspecting user in the context of the affected site. Is there any way to mitigate this vulnerability (I suspect anyone using Tomcat is going to see the same thing)? What vulnerability? I don't see any evidence (no Tomcat version, no CVE reference, no PoC) to back up the claim of a vulnerability. Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org . Mark, Thanks for the reply. I agree, I didn't see any references, just the blanket statement so I posted it to see if anyone else had bumped into this. I guess I will just challenge them to provide some evidence and see where it goes. Thanks, Carl - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat cross-site scripting vulnerability
On 7/4/2014 9:46 AM, Vijendra Pachoriya wrote: Which version of tomcat you are using ?? Either upgrade to tomcat 7 or add this to your tomcat context.xml Context useHttpOnly=true Regards, Vijendra -Original Message- From: Radha Krishna Meduri -X (radmedur - HCL TECHNOLOGIES LIMITED at Cisco) [mailto:radme...@cisco.com] Sent: 04 July 2014 18:45 To: Tomcat Users List Subject: RE: Tomcat cross-site scripting vulnerability I think application needs to take care of CSRF. -Original Message- From: carl [mailto:c...@etrak-plus.com] Sent: Friday, July 04, 2014 6:43 PM To: users@tomcat.apache.org Subject: Tomcat cross-site scripting vulnerability Our latest PCI scan using the Saint scanner shows the following: 404 Error Page Cross Site Scripting Vulnerability 12/21/09 Apache Tomcat is prone to a cross-site scripting vulnerability because it fails to properly sanitize user-supplied input. An attacker may leverage this issue to execute arbitrary script code in the browser of an unsuspecting user in the context of the affected site. Is there any way to mitigate this vulnerability (I suspect anyone using Tomcat is going to see the same thing)? Thanks, Carl - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org . Thanks for the suggestion. I am using 7 and am upgrading to the latest version. Thanks, Carl - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Curious difference in connection behaviour on database side DBCP vs. JDBC?
Thanks for taking the time Daniel,
It is very hard to explain the problem since, and it was also stupid of me
to not include the fact that I have tried all kinds of similar combinations
of configuration in context.xml. With botch dbcp and jdbc pools
The behaviour persists. For example these are one version of config. I have
tested.
DBCP: behaviour is absent and all i well
Resource name=database1
auth=Container
maxActive=50
maxIdle=8
minIdle=2
initialSize=0
username='
password='
driverClassName=com.inet.tds.TdsDriver
type=javax.sql.DataSource
defaultTransactionIsolation=READ_UNCOMMITTED
url=jdbc:inetdae7://devdb12:1433/database1_dev
testOnBorrow=true
validationQuery=SELECT 1
timeBetweenEvictionRunsMillis=1
removeAbandoned=true
removeAbandonedTimeout=600
maxWait=1/
JDBC: I see the weird behaviour and my DBA is angry
Resource name=database1
auth=Container
maxActive=50
maxIdle=10
minIdle=2
initialSize=0
username='
password='
driverClassName=com.inet.tds.TdsDriver
type=javax.sql.DataSource
factory=org.apache.tomcat.jdbc.pool.DataSourceFactory
defaultTransactionIsolation=READ_UNCOMMITTED
defaultAutoCommit=true
url=jdbc:inetdae7://devdb12:1433/database1_dev
testOnBorrow=true
validationQuery=SELECT 1
timeBetweenEvictionRunsMillis=1
removeAbandoned=true
removeAbandonedTimeout=600
maxWait=1/
The behaviour applies to ALL queries/statements from the application.
I have here an example of the way we close from the application, (the devs
have named it dispose). From my untrained non java dev eye we do not seem
to be doing statement.Close(); and Im curious if that might be the issue?
If so, why does DBCP handle it nicely and not JDBC?
public void dispose() {
if (connection != null) {
try {
if (!connection.isClosed()) {
// If autoCommit is false, we are most likely using
transactions. A rollback will end the transaction
// properly even if a pool treats all actual
connections to the db as single long transactions.
// Examine the connection directly instead of relying
on ConnectionManager attribute.
if (ROLLBACK_ON_CLOSE !connection.getAutoCommit()) {
connection.rollback();
}
// Close the connection
connection.close();
if (traceOpenedConnections) {
timeConnectionClosed = System.currentTimeMillis();
}
}
} catch (java.sql.SQLException sqle) {
sqle.printStackTrace();
}
this.connection = null;
}
// Deregister this ConnectionManger
if (traceOpenedConnections) {
deregister(this);
}
}
TNTUnilab delenda est
---
Carl Boberg
Operations
Memnon Networks AB
Tegnérgatan 34, SE-113 59 Stockholm
Mobile: +46(0)70 467 27 12
www.memnonnetworks.com
On 18 November 2013 18:23, Daniel Mikusa dmik...@gopivotal.com wrote:
On Nov 18, 2013, at 9:48 AM, Carl Boberg carl.bob...@memnonnetworks.com
wrote:
Hello,
We have recently migrated from dbcp pool to the newer tomcat-jdbc pool.
As
I understand, it is supposed to be almost a drop in replacement for
dbcp.
*Almost* is the key word here. It's very similar, but there are
differences (typically with default values). Most of the time these don't
matter, but occasionally you'll bump into them.
Everything works great except once small thing. Its a bit difficult for
me
to explain but our DBA is really annoyed by it...
With the new jdbc configuration, idle connections never really turn into
normal idle on the database side.
The DBS looks at the connections on the db side with sp_whoisactive
(excellent improvement of the sp_whoX procedures by Adam Machanic). It
should just show the statement/queries that are actually running. After
that the connection should be empty and idle and thus not be visible...
But with the new JDBC pool it shows the last statement/query executed in
the connection and it only changes when a new query comes from the
application (see below)* I therefore know that the connection gets reused
as an idle connection should but this is not the expected behaviour. It
should not linger on th db server like it does..
Sorry, I'm not certain what you mean here. I don't use MSSQL, but perhaps
someone else does and can comment.
I automatically would assume there is something in the code not closing
statements
Curious difference in connection behaviour on database side DBCP vs. JDBC?
Hello, We have recently migrated from dbcp pool to the newer tomcat-jdbc pool. As I understand, it is supposed to be almost a drop in replacement for dbcp. Everything works great except once small thing. Its a bit difficult for me to explain but our DBA is really annoyed by it... With the new jdbc configuration, idle connections never really turn into normal idle on the database side. The DBS looks at the connections on the db side with sp_whoisactive (excellent improvement of the sp_whoX procedures by Adam Machanic). It should just show the statement/queries that are actually running. After that the connection should be empty and idle and thus not be visible... But with the new JDBC pool it shows the last statement/query executed in the connection and it only changes when a new query comes from the application (see below)* I therefore know that the connection gets reused as an idle connection should but this is not the expected behaviour. It should not linger on th db server like it does.. I automatically would assume there is something in the code not closing statements properly and/or connection leak or some such... BUT When using the old dbcp pool configuration all connections behaved as expected .. .statement/query get run and the connection turns to a normal idle connection with no information regarding earlier queries/statements... therefore it never shows up in sp_whoisactive which is expected. For example: rebooting the application an not doing anything (test env. no users no nothing) if I execute sp_whoisactive I currently see the default startup query from the app * 00 00:31:59.793 333 ?query -- update Account set loggedIn = 0 where loggedIn = 1 --? Sleeping Now this is a short simple fast query and it shouldn't show here but its shows as been hanging around for half an hour. It doesn't hang around like this if I revert to the old dbcp configuration... - the issue Am I missing something fundamental in expected behaviour or should I just go shout at the devs (who will only say it worked with the old pool so it should work the same now)... We are using tomcat 6.32 / jdk1.6.0_26 with MS SQL server 2012 Old dbcp conf, everything works fully as expected Resource name=database1 auth=Container maxActive=50 maxIdle=8 username=* password= driverClassName=com.inet.tds.TdsDriver type=javax.sql.DataSource defaultTransactionIsolation=READ_UNCOMMITTED url=jdbc:inetdae7://sqlserver12:1433/database1_dev timeBetweenEvictionRunsMillis=30 removeAbandoned=true removeAbandonedTimeout=600 maxWait=1/ JDBC configuration with tomca-jdbc.jar taken from tomcat7.42, has idle connections with statement/query info in them, so that the dba never really sees them as fully idle Resource name=database1 auth=Container maxActive=50 maxIdle=10 minIdle=2 initialSize=2 username=* password=* driverClassName=com.inet.tds.TdsDriver type=javax.sql.DataSource factory=org.apache.tomcat.jdbc.pool.DataSourceFactory defaultTransactionIsolation=READ_UNCOMMITTED jdbcInterceptors=ConnectionState;StatementFinalizer defaultAutoCommit=true url=jdbc:inetdae7://sqlserver12:1433/database1_dev testOnBorrow=true validationQuery=SELECT 1 timeBetweenEvictionRunsMillis=1 removeAbandoned=true removeAbandonedTimeout=1800 maxWait=1/ I have googled, read documentation, browsed forums but never seen anyone having an issue like this... and I admit, its nothing major. I'm just very curious about the source to this difference in behaviour (and can I do something configuration wise to resolve it?). Best regards --- Carl Boberg
lost session in Tomcat 7.040 and IE8
I have Tomcat 7.0.26 running on Window7 Pro. I also have Tomcat 7.0.40 running on a Windows 7 Home Premium. Both have the same website. (Obviously, I'm doing some testing.) In the website, a user logs on and the user ID is kept in the session. In one of the JSP pages I have some JavaScript attached to an html button input type=button name= value=blah blah blah onclick=window.location='/MySite/MyAction.do' (I'm using Struts.) Now, here is were it gets strange... During testing, I found that IE8 and IE9 both run fine against Tomcat 7.0.26. By that I mean, after the user logs on, the user ID is kept in the session. After navigating around the site, if the user then clicks on the above button, the Struts Action class MyAction.do is able to find the user ID in the session. The same is true of IE9 against Tomcat 7.0.40. But if I do the above with IE8 against the site on Tomcat 7.0.40, the user ID in the session is empty. To summarize, | IE8 | IE9 --- Tomcat 7.0.26 | ok | ok --- Tomcat 7.0.40 |fail | ok --- Any ideas where to start looking? - Carl Dreher - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat dies suddenly
Interesting and thanks for your response. We switched back to VM version 1.6.0_07 (64 bit on Slackware) and everything was fine. For several unrelated reasons, we switched to Tomcat version 6.0.35 and JVM (Oracle) version 1.6.0_16 and all has been fine for a month or so. The cause remains unknown but seemingly gone now. Thanks, Carl On May 16, 2012, at 1:53 PM, slayer12 wrote: I was facing a problem with similar symptoms - Tomcat 7 (on centos 5, 64 bit, 12 GB RAM) was crashing without anything being logged in catalina.out In case of JVM crash, there is supposed to be some hs_err_pid.log file but that was also not getting generated. Neither were there any OutOfMemory messages in /var/log/messages Turns out that my code was going into an infinite loop because I was using Calendar.before(Date) and this method (from java.util.calandar) stupidly takes any object as a parameter but returns false if the passed object is anything other than and instance of Calendar. Sun knows about this, but in their infinite wisdom had closed 2 java bugs related to this without resolving anything !! http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=4682471 http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=4738710 - just my 2 bits -- View this message in context: http://tomcat.10.n6.nabble.com/Tomcat-dies-suddenly-tp2136492p4980853.html Sent from the Tomcat - User mailing list archive at Nabble.com. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat silently dies
David, Thanks, I will try that. Carl On Mar 4, 2012, at 7:45 PM, David Kerber wrote: On 3/4/2012 7:31 PM, Carl Kabbe wrote: Recap: OS Slackware 13.x 64bit Tomcat 6.0.24 Two different servers… one with 16 GB and one with 8GB Heap 2GB, PermGen 300MB Tried different versions of the (Oracle) JVM … tried 1.6.0_24 and 1.6.0_31 The current solution: Has been stable using JVM version 1.6.0_7 for the last week (we were going down once or twice a day.) Discussion: Two years ago, I faced this same issue and Chuck suggested I revert to 1.6.0_7. He said there was a change in 1.6.0_10 that may have caused the problem… I do not know what that was but he appeared to be correct. I reverted to 1.6.0_7 and the system was stable for two years until I tried more recent JVM's. At this point, I am looking for suggestions as it clearly has something to do with the JVM (I don't want to stay on 1.6.0_7 forever) but I have no idea how to duplicate or troubleshoot the problem. Thanks, Carl Update to a later Tomcat? Maybe there's some interaction between it and the JVM... just a swag... - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Tomcat silently dies
Recap: OS Slackware 13.x 64bit Tomcat 6.0.24 Two different servers… one with 16 GB and one with 8GB Heap2GB, PermGen 300MB Tried different versions of the (Oracle) JVM … tried 1.6.0_24 and 1.6.0_31 The current solution: Has been stable using JVM version 1.6.0_7 for the last week (we were going down once or twice a day.) Discussion: Two years ago, I faced this same issue and Chuck suggested I revert to 1.6.0_7. He said there was a change in 1.6.0_10 that may have caused the problem… I do not know what that was but he appeared to be correct. I reverted to 1.6.0_7 and the system was stable for two years until I tried more recent JVM's. At this point, I am looking for suggestions as it clearly has something to do with the JVM (I don't want to stay on 1.6.0_7 forever) but I have no idea how to duplicate or troubleshoot the problem. Thanks, Carl
Re: Tomcat suddenly dies
Chuck and Chris, Thanks for your replies. Below is some information to your questions/suggestions: Check the kernel logs (e.g., /var/log/messages, /var/log/warn), not just the Tomcat ones. Also, look for a JVM dump file (hs_err_pid*.log) I have and there is nothing in the messages file except accesses granted to specific workstations coming in on ssh and sync'ing to a time server. Neither of these have times that correspond to the crashes. There are no hs_err_* files anywhere on the servers. Smells a lot like OOM killer. Carl, you say you have a 2GiB heap. Are you using 32-bit or 64-bit JVM? What about other large-memory processes on the same boxes? Do you have other JVMs running or a database, etc.? Does the JVM die on any kind of schedule? We are running 64 bit OS's (Slackware 13.x, the latest version.) There are two other applications running on each of the boxes: 1) the Apache James email server (localhost SMTP only) and 2) a small application that serves reports. They are both very small (the current server shows 11GB+ free memory) and always survive theTomcat crashes. These servers are only used for Tomcat (and the related James and report serving app.) Not on a timed schedule but usually during high traffic periods (usually, but not always, as with last Friday.) Thanks, Carl On Feb 28, 2012, at 11:13 AM, Christopher Schultz wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Chuck, On 2/27/12 9:01 PM, Caldarale, Charles R wrote: From: Carl Kabbe [mailto:c...@etrak-plus.com] Subject: Tomcat suddenly dies Starting about a month ago, Tomcat would suddenly fail, no heap dump, no indication of trouble in the log, no indication the JVM crashed Check the kernel logs (e.g., /var/log/messages, /var/log/warn), not just the Tomcat ones. Also, look for a JVM dump file (hs_err_pid*.log), possibly in the current directory of what was the Tomcat process. +1 Smells a lot like OOM killer. Carl, you say you have a 2GiB heap. Are you using 32-bit or 64-bit JVM? What about other large-memory processes on the same boxes? Do you have other JVMs running or a database, etc.? Does the JVM die on any kind of schedule? - -chris -BEGIN PGP SIGNATURE- Version: GnuPG/MacGPG2 v2.0.17 (Darwin) Comment: GPGTools - http://gpgtools.org Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk9M/RcACgkQ9CaO5/Lv0PDtmQCgmnkl/KExQjSradWMoIvxEM+Y AWsAmgJudFWslkrIZLv7uID+uWUu9HZs =lUVS -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat suddenly dies
Ranier, Thanks for your response and thoughts. Are there normal shutdown messages in the Tomcat logs? No, the catalina.out log has our messages but none of the normal shutdown stuff (we see the shutdown messaging every day when we push changes and restart Tomcat at 4:00AM.) Thanks, Carl On Feb 28, 2012, at 2:00 PM, Rainer Jung wrote: On 28.02.2012 19:47, Carl Kabbe wrote: Chuck and Chris, Thanks for your replies. Below is some information to your questions/suggestions: Check the kernel logs (e.g., /var/log/messages, /var/log/warn), not just the Tomcat ones. Also, look for a JVM dump file (hs_err_pid*.log) I have and there is nothing in the messages file except accesses granted to specific workstations coming in on ssh and sync'ing to a time server. Neither of these have times that correspond to the crashes. There are no hs_err_* files anywhere on the servers. Smells a lot like OOM killer. Carl, you say you have a 2GiB heap. Are you using 32-bit or 64-bit JVM? What about other large-memory processes on the same boxes? Do you have other JVMs running or a database, etc.? Does the JVM die on any kind of schedule? We are running 64 bit OS's (Slackware 13.x, the latest version.) There are two other applications running on each of the boxes: 1) the Apache James email server (localhost SMTP only) and 2) a small application that serves reports. They are both very small (the current server shows 11GB+ free memory) and always survive theTomcat crashes. These servers are only used for Tomcat (and the related James and report serving app.) Not on a timed schedule but usually during high traffic periods (usually, but not always, as with last Friday.) Are there normal shutdown messages in the Tomcat logs? Regards, Rainer - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat suddenly dies
Dan, Thanks for the response. Are you accessing any native code from Java? For example using JNI / JNA or a third party module which requires you to set java.library.path? No. Thanks, Carl On Feb 28, 2012, at 2:35 PM, Daniel Mikusa wrote: On Tue, 2012-02-28 at 11:09 -0800, Carl Kabbe wrote: Ranier, Thanks for your response and thoughts. Are there normal shutdown messages in the Tomcat logs? No, the catalina.out log has our messages but none of the normal shutdown stuff (we see the shutdown messaging every day when we push changes and restart Tomcat at 4:00AM.) Are you accessing any native code from Java? For example using JNI / JNA or a third party module which requires you to set java.library.path? Dan Thanks, Carl On Feb 28, 2012, at 2:00 PM, Rainer Jung wrote: On 28.02.2012 19:47, Carl Kabbe wrote: Chuck and Chris, Thanks for your replies. Below is some information to your questions/suggestions: Check the kernel logs (e.g., /var/log/messages, /var/log/warn), not just the Tomcat ones. Also, look for a JVM dump file (hs_err_pid*.log) I have and there is nothing in the messages file except accesses granted to specific workstations coming in on ssh and sync'ing to a time server. Neither of these have times that correspond to the crashes. There are no hs_err_* files anywhere on the servers. Smells a lot like OOM killer. Carl, you say you have a 2GiB heap. Are you using 32-bit or 64-bit JVM? What about other large-memory processes on the same boxes? Do you have other JVMs running or a database, etc.? Does the JVM die on any kind of schedule? We are running 64 bit OS's (Slackware 13.x, the latest version.) There are two other applications running on each of the boxes: 1) the Apache James email server (localhost SMTP only) and 2) a small application that serves reports. They are both very small (the current server shows 11GB+ free memory) and always survive theTomcat crashes. These servers are only used for Tomcat (and the related James and report serving app.) Not on a timed schedule but usually during high traffic periods (usually, but not always, as with last Friday.) Are there normal shutdown messages in the Tomcat logs? Regards, Rainer - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Tomcat suddenly dies
I have run Tomcat in two environments and both produced the same results. Server 1: OS - Slackware Linux 13.x 64 bit JVM - 1.6.0_31 Tomcat - 6.0.24 Server memory 16GB Heap 2GB, PermGen 300M Server 2: OS - Slackware 13.x 64 bit JVM - 1.6.0_31 Tomcat - 7.0.14 Server memory 8GB Heap 2GB, PermGen 300M The history: Two years ago, I was running the application on JVM 1.6.0_18 and having a similar or the same problem and Charles suggested I go back to 1.6.0_7. I did and the app ran pretty flawlessly (except for screwups in the code changes.) We have been adding a number of clients so I am moving toward a clustered environment and trying to update the JVM, Tomcat, etc. The problem: Starting about a month ago, Tomcat would suddenly fail, no heap dump, no indication of trouble in the log, no indication the JVM crashed (JVisualVM continues to hum along), nothing (that I can find.) At first, we thought it was load related as our heaviest loads are in the early afternoon and the crashes were consistently 1-2:00PM. Then, last Friday, it crashed around 4:00PM when there was relatively light load. As nearly as we can tell, there is no slowness or other indicator the failure is coming… the system just hums along and bam, gone. The garbage collection and cpu utilization look pretty normal (garbage collection is the typical saw edge and cpu utilization is 0-20% with an occasional short duration spike.) I am going back to 1.6.0_7 on server 1 tomorrow. I was thinking the only way to potentially get to the bottom of this would be to take frequent heap dumps between 1:00 and 2:00PM but, while they may be interesting, I am not certain they will show anything useful as the system shows no signs of strain before dying. Anyone have any ideas? Thanks, Carl - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Why is the tomcat welcome page showing?
Hey all, In server.xml I have a Host section set up with a Context that has a path set to . I see in the logs that the webapp assigned to that path is indeed executing when I hit '/'. However the browser shows the Tomcat homepage. If you're seeing this page via a web browser, it means you've setup Tomcat successfully. Congratulations! Why is my webapp not showing in my browser? Do I have to set up a DocumentIndex type parameter like in Apache? How do I disable the ROOT context? Thanks, Carl Furst smime.p7s Description: S/MIME cryptographic signature ** MLB.com: Where Baseball is Always On - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Why is the tomcat welcome page showing?
To follow up on this.. I removed the ROOT folder from the webapps.. and it still shows up even when I have re-defined the default Context. Where does the welcome page get rendered? Thanks, Carl Furst -Original Message- From: Furst, Carl [mailto:carl.fu...@mlb.com] Sent: Monday, August 15, 2011 4:49 PM To: users@tomcat.apache.org Subject: Why is the tomcat welcome page showing? Hey all, In server.xml I have a Host section set up with a Context that has a path set to . I see in the logs that the webapp assigned to that path is indeed executing when I hit '/'. However the browser shows the Tomcat homepage. If you're seeing this page via a web browser, it means you've setup Tomcat successfully. Congratulations! Why is my webapp not showing in my browser? Do I have to set up a DocumentIndex type parameter like in Apache? How do I disable the ROOT context? Thanks, Carl Furst smime.p7s Description: S/MIME cryptographic signature ** MLB.com: Where Baseball is Always On - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Why is the tomcat welcome page showing?
Hi Charles, Thanks for taking the time to answer. I'm using Tomcat 6.0 on Solaris I understand it's not ideal, however, I have to work this way because of a framework built for a different platform. The deployment mechanisms are such that they don't deploy to ROOT. Anyway with you're inspiring words of webapp isn't being deployed properly got me going and rebuilding, killing the jdk and starting up again. Something got cleared and it's working again. Very much obliged, Carl Furst -Original Message- From: Caldarale, Charles R [mailto:chuck.caldar...@unisys.com] Sent: Monday, August 15, 2011 5:09 PM To: Tomcat Users List Subject: RE: Why is the tomcat welcome page showing? From: Furst, Carl [mailto:carl.fu...@mlb.com] Subject: Why is the tomcat welcome page showing? In server.xml I have a Host section set up with a Context that has a path set to . Unfortunate that you choose to use the least desirable method of setting the default webapp. I see in the logs that the webapp assigned to that path is indeed executing when I hit '/'. However the browser shows the Tomcat homepage. Possibly browser caching, but also possible that your webapp isn't being deployed properly. How do I disable the ROOT context? You can't - there must always be a ROOT context; you have just chosen a poor (and conflicting) way to specify it. What you should be doing: 1) Tell us the version of Tomcat you're using. 2) Stop Tomcat. 3) Remove the existing ROOT directory from the Host appBase directory. 4) Rename your webapp to ROOT (or ROOT.war, if appropriate). 5) Remove the Context element from server.xml. 6) Remove the contents of Tomcat's work and temp directories. 7) Remove the ROOT.xml file (if it exists) from conf/Catalina/[host]. 8) Restart Tomcat. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org smime.p7s Description: S/MIME cryptographic signature ** MLB.com: Where Baseball is Always On - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Can't configure webapps to disk partition
Hello and thank you in advance for any suggestions you may have. I am running Tomcat 6.0.24 (service) on a Windows 2003 server hosting very simple JSP pages. I am trying to make a configuration change to server.xml to change the webapps directory to a different disk partition as below but I cannot get it to work From: Host name=localhost appBase=webapps unpackWARs=true autoDeploy=true xmlValidation=false xmlNamespaceAware=false To: Host name=localhost appBase=E:\prod\applications unpackWARs=true autoDeploy=true xmlValidation=false xmlNamespaceAware=false I have had success changing the webapps directory to various locations on the C: Drive but when I try to use the E: partition, I receive the page cannot be found message. The reason I need to do this is because the path (E:\prod\applications) is the target source code deployment path required by my employer. I have found several posts in the forum that suggest variations on the syntax as noted below but still, I have not been able to get anything to work. The WEB-INF and META-INF as well as web.xml are all present in the E:\prod\applications\project name folder and I have restarted the Tomcat service after my changes. I have tried many variations of these: E://prod//applications \E:\prod\applications Thank you, Carl - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat died on java.lang.OutOfMemoryError: requested 2147483664 bytes for Chunk::new. Out of swap space? message
I am the person who reported occasional but persistent abrupt terminations of the 1.6 JVM on levels 6u10 and above. I did go back to 6u7 and the application has run without a burp for three months. I had tested 6u18/6u19, both of which produced the same result. I am getting ready to start testing 6u20 as I would like to keep current so I don't have any security issues hanging out. Thanks, Carl - Original Message - From: Caldarale, Charles R chuck.caldar...@unisys.com To: Tomcat Users List users@tomcat.apache.org Sent: Wednesday, May 26, 2010 8:00 AM Subject: RE: Tomcat died on java.lang.OutOfMemoryError: requested 2147483664 bytes for Chunk::new. Out of swap space? message From: Caldarale, Charles R Subject: RE: Tomcat died on java.lang.OutOfMemoryError: requested 2147483664 bytes for Chunk::new. Out of swap space? message The current JVM version is 6u20, so you might want to try running with that before filing a bug report or expanding the swap file. Should also mention that another Tomcat admin reported occasional but persistent abrupt terminations of the 1.6 JVM on levels 6u10 and above, but went back to 6u7 and that appeared to rectify the problem. I think the highest level he had tested was 6u17, but I'm not positive. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat Shutdown suddenly / random
I thought thta a System.exit call would kill the JVM and would therefore not show the clean shutdown in the logs that the OP is seeing... am I wrong about System.exit? Thanks, Carl - Original Message - From: Christopher Schultz ch...@christopherschultz.net To: Tomcat Users List users@tomcat.apache.org Sent: Friday, April 16, 2010 3:58 PM Subject: Re: Tomcat Shutdown suddenly / random -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Harry, On 4/16/2010 2:35 PM, Harry Metske wrote: could it be that something is sending your tomcat process a TERM signal, logfiles in /var/log might tell something ? or one of your applications issues a System.exit() under certain circumstances ? +1 I'm not sure what the best way is to catch TERM signals. I guess you could write some JNI code to install a signal handler that logs the signal and details (if available) about the process that sent the signal. I don't believe Tomcat has any System.exit calls in it, so you could grep your code looking for such calls. - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkvIz4IACgkQ9CaO5/Lv0PAT4QCgtsZhK/DtHhS8KVjYUhCA2mdG dVwAn1DOyYGJLIfV5hBl1GWaTF8CZUO3 =ASpm -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: [OT] Re: jvm exits without trace
Dan, 6u18 did not work for us, crashed with the same regularity as 6u17. However, 6u7 has been running for two weeks without a failure (the others would fail between 15 minutes and 10 days runtime.) Thanks, Carl - Original Message - From: Dan Armbrust daniel.armbrust.l...@gmail.com To: Tomcat Users List users@tomcat.apache.org Sent: Monday, March 22, 2010 1:17 PM Subject: Re: [OT] Re: jvm exits without trace On Tue, Mar 16, 2010 at 4:46 PM, Carl c...@etrak-plus.com wrote: My approach is to get something (a JVM) that works and then gradually change until it breaks. Then, I know what is causing the problem. To date, I haven't been able to get a JVM that works. I have had a lot of issues finding stable JVMs since I moved our software from 1.5 to 1.6... 1.6 has been a mess for ages under our workload on CentOS. Code that ran fine under 1.5 would segfault in all sorts of random places on 1.6. I even tried the 1.7 openJDK early builds... they were even worse. I was pleased to see that the most recent release http://java.sun.com/javase/6/webnotes/6u18.html contains _tons_ of bug fixes, including lots of crash fixes. Seriously... the last few releases have contained only a handful of fixes... this release has hundreds. I'm testing it now, and so far, it looks promising. This may be the first 1.6 release that I've found that doesn't crash with my workload. Unfortunately, I've never been able to pin down a sequence of events that would cause the crash on demand, so its hard for me to verify that things are finally fixed, other than doing long term load testing and waiting for the crashes to (hopefully) not happen. Dan - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: [OT] Re: jvm exits without trace
George, I tried OpenSuSE with 6u17 and the results were about the same as Slackware 64 so the implication is that there is something in my app that is triggering the seg fault. My app is mostly jsp's running against MySQL with some AJAX and Flash (communicating through a servlet.) I do not use any native code nor do I use AJP. I do use SSL for everything. It has also failed on different hardware although the OS was the same. The argument against the problem being in my app is that the app runs fine (has for several years) on 32 bit Slackware and it now seems to be running fine on 6u7 (on Slackware 64.) Now that I appear to have a stable system that I can always go back to, I would like to get to the root cause. My thought was that I could get the source code, compile it on the server and then wait for it to fail with a core file that could give me the answer (maybe.) However, I can only find source code for the 'open' fork which may or may not be the same so I am stymied again. I guess I just have to keep stabbing in the dark with the new JVM's until one happens to work... not very scientific. Thanks, Carl - Original Message - From: George Sexton geor...@mhsoftware.com To: 'Tomcat Users List' users@tomcat.apache.org Sent: Monday, March 22, 2010 2:48 PM Subject: RE: [OT] Re: jvm exits without trace -Original Message- From: Dan Armbrust [mailto:daniel.armbrust.l...@gmail.com] Sent: Monday, March 22, 2010 12:17 PM To: Tomcat Users List Subject: Re: [OT] Re: jvm exits without trace On Tue, Mar 16, 2010 at 4:46 PM, Carl c...@etrak-plus.com wrote: My approach is to get something (a JVM) that works and then gradually change until it breaks. Then, I know what is causing the problem. To date, I haven't been able to get a JVM that works. I have had a lot of issues finding stable JVMs since I moved our software from 1.5 to 1.6... 1.6 has been a mess for ages under our workload on CentOS. Code that ran fine under 1.5 would segfault in all sorts of random places on 1.6. I even tried the 1.7 openJDK early builds... they were even worse. I was pleased to see that the most recent release http://java.sun.com/javase/6/webnotes/6u18.html contains _tons_ of bug fixes, including lots of crash fixes. Seriously... the last few releases have contained only a handful of fixes... this release has hundreds. I'm running 1.6.0_18 under OpenSuSE 11.0-11.2 and I've only had one problem. It looks like a GLIBC error related to IPV6. I disabled IPV6 on the machine and it's been rock solid since. I don't use native connectors, or AJP. I do about 900,000 hits with 4GB/Day spread across 3 servers and it's just rock steady. I run the tomcat instances for 2-3 weeks each before re-starting. For each machine, that's about 6 million hits with about 40 GB of transfer. George Sexton MH Software, Inc. 303 438-9585 www.mhsoftware.com - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat dies suddenly
The application has been running for 14 days without failing using the Sun JVM 1.6.0_7 (per Chuck's suggestion... thanks Chuck.) However, we all know that is just a work around. To find the casue, my plan is to bring the latest JVM source code down and build the JVM on the server with debugging information so that the core file can tell us where it failed and the condition at failure. Is ther as easier path? Thanks, Carl - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: jvm exits without trace
Taylan, I have had a similar problem that is yet unsolved (see the thread 'Tomcat dies suddenly'.) In my case, the death left a core file which showed the JVM stopped with a seg fault. A week ago yesterday, we switched to the Sun 1.6.0_7 JVM (from 1.6.0_17 and 1.6.0_17) (Chuck suggested this) and so far, it is running even though we have had loads and usages similar to those that caused crashes in the past. Therefore, you might consider trying that JVM. Hope I haven't jinxed myself by saying it is still up. Thanks, Carl - Original Message - From: Taylan Develioglu tdevelio...@ebuddy.com To: Tomcat Users List users@tomcat.apache.org Sent: Tuesday, March 16, 2010 7:41 AM Subject: Re: jvm exits without trace With parent I meant the main JVM process as opposed to forked processes or threads, sorry to confuse you there. Stracing the threads generates too much data to store so I had to settle with the parent process. To answer your other questions. The code is 100% pure java, why it causes this messy crash is still unclear but development is working to figure it out. I'll follow up when we find out more, but I'm not sure if we're likely to dig into the root cause, working around it is more of a priority right now than debugging the jvm. On Mon, 2010-03-15 at 17:08 +0100, Christopher Schultz wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Taylan, On 3/15/2010 10:19 AM, Taylan Develioglu wrote: The cause for the crashes was in our own application code, we're currently investigating the exact reason. Yeah, I'd like to second Chuck's question: was it native code? A strace of the parent process shows killed by sigsegv, why or how this can happen is still unclear. So, the parent was being killed? What was the parent of the JVM? Thanks to everyone that gave their assistance. Definitely follow-up to let us all know what you've uncovered... this was certainly a weird situation. - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkueW4wACgkQ9CaO5/Lv0PAdhgCfa32vlcsMI5ELCNcLSjjV+S/o FZEAnjvjXgAwxjejTXexGO//89TyeF+r =BPtZ -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: [OT] Re: jvm exits without trace
My approach is to get something (a JVM) that works and then gradually change until it breaks. Then, I know what is causing the problem. To date, I haven't been able to get a JVM that works. In my case, it might be something in my application that is causing the crash on 64 bit Slackware as there are many people running 64 bit on Linux without a problem. If it is my application, then any 64 bit JVM I throw at it should crash. On the other hand, if the 1.6.0_7 JVM works, then it is likely a bug in one of the changes that brings the JVM to the current version. Time will tell. Thanks, Carl - Original Message - From: André Warnier a...@ice-sa.com To: Tomcat Users List users@tomcat.apache.org Sent: Tuesday, March 16, 2010 3:20 PM Subject: Re: [OT] Re: jvm exits without trace Ognjen Blagojevic wrote: Hi, Tomcat dies suddenly thread was exciting almost as Prison Break TV series. I couldn't wait to find out what would be solution to the problem, and I must admit that just downgrading to lower sub-sub-sub version JVM left me a bit disappointed. :) +1 It sounds quite like the standard tech support solution for Windows problems : de-install, re-install; and if that does not help, push the reset button. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: jvm exits without trace
Taylan, I am currently trying JVM 1.6.0_7 per Chuck's suggestion and, so far (4 days), it is working. I started down the IBM JVM path but have abandoned that for now due to difficulties with the SSL implementation (somne browsers would work and some wouldn't with seemingly the same setup.) Thanks, Carl - Original Message - From: Taylan Develioglu tdevelio...@ebuddy.com To: Tomcat Users List users@tomcat.apache.org Sent: Thursday, March 11, 2010 6:13 AM Subject: Re: jvm exits without trace a different kernel did not help either... On Thu, 2010-03-11 at 11:37 +0100, Taylan Develioglu wrote: Changing to JIO didn't help, the silent crashes continue. I'm changing kernel versions now. On Fri, 2010-03-05 at 10:45 +0100, Taylan Develioglu wrote: It's performing rather poorly performance wise, compared to the apr connector. The number of threads required to handle the requests has gone up significantly over the board. Stability wise, I don't have complaints yet. I'm keeping my fingers crossed. On Fri, 2010-03-05 at 10:09 +0100, Pid wrote: On 05/03/2010 08:41, Taylan Develioglu wrote: Pid, that would assume we had a working 1.6.10 version before that we replaced. That it would. We've run 1.6.10 upwards succesfully for a very long time. So I don't see the point in doing this. I must have missed that. How is the HTTP connector performing? p On Wed, 2010-03-03 at 12:00 +0100, Pid wrote: On 03/03/2010 09:11, Taylan Develioglu wrote: Downgrading to 1.6.0_16 did not help. I'm replacing the apr connector with http now. As Chuck mentioned in the other thread, significant changes occurred at 1.6.10, so trying the release before (1.6.7) might be necessary to establish a better determination. p On Wed, 2010-02-24 at 14:52 +0100, Carl wrote: Taylan, The failures we've seen are in anywhere between 8 hours to a week of runtime. The timing of the failures seems similar. We have also had failures with hotspot error files (hs_err) present, and the cause specified was indeed SIGSEGV indicating a page fault. I have never seen any hs_* files but have seen core files where strace showed the jvm stopped on a seg fault. We also use jdk 1.6.0_18, I'm downgrading the machines to 1.6.0_16 when the situation allows (during regular updates of the application, or a crash) to see if that helps. I have used jdk 1.6.0_17 and 1.6.0_18 with the same results... have not tried 1.6.0_16. Please post your results of this trial. Running tomcat on the foreground might show something, but then again I could be waiting for a month for it to happen. Yes, this has been part of my problem as anytime we change something, we have to wait a week for the server to fail. In one sense, I am fortunate that I have a little more flexibility than you. I have two servers (different hardware) but only need one in service at a time. Therefore, I always have one server I can test ideas on although I have never been able to develop a meaningful stress test, i.e., the only way I can test a change is to put it in production. Thanks, Carl - Original Message - From: Taylan Develioglutdevelio...@ebuddy.com To: Tomcat Users Listusers@tomcat.apache.org Sent: Wednesday, February 24, 2010 8:31 AM Subject: Re: jvm exits without trace Hello Carl, The failures we've seen are in anywhere between 8 hours to a week of runtime. Most of them have (still) been running for almost a month without failure. There are ~100 machines. From the top of my head, I think we've had about 10+ failures now. We have also had failures with hotspot error files (hs_err) present, and the cause specified was indeed SIGSEGV indicating a page fault. But I don't know if the two are related. We also use jdk 1.6.0_18, I'm downgrading the machines to 1.6.0_16 when the situation allows (during regular updates of the application, or a crash) to see if that helps. It might be useful to note that the failures happen with tomcat 6.0.20 as well as 6.0.24. As far as load concerns, I haven't had a failure on an idle machines. The machines are well loaded, but only at a fraction limit in regards to load and cpu utilization. Most memory is commited to tomcat, where a 24G machine would have 18G allocated to heap, 128M to permgen and some unspecified amount would get used by jni for apr. About 4G remains free after calculating taking into account the jvm itsself. A 16G machine would have 12G allocated to the heap. Besides the fact that our apps heavily use nio and mina I wouldn't say there's anything else noteworthy. There can be anywhere up to 1
SSL with IBM JVM
I am trying the IBM JVM as a possible cure for the problem 'Tomcat dies suddenly'. A quick recap: New Dell T110 Slackware 13 - 64 bit Tomcat 6.0.24 The problem: Tomcat would die suddenly without any entry in any log but would leave a core file that indicated the JVM exited with a seg fault. I had been using the Sun JVM 1.6.0_18. I have tried 1.6.0_16 (because Taylan said his system started a similar problem when he went from 1.6.0_16 to 1.6.0_18) but not 1.6.0_9 (as Chuck suggested.) Switched to the latest IBM JVM last Friday. We run https for all applications as we deal mostly with children's data. About 10% of the users experienced problems with accessing the site. Both IE and Firefox caused problems. Switching them to http eliminated the problem but we can not run that way on a regular basis. Here is what we have done: IE - IE 6 simply doesn't work. If IE 8 is installed, removing it and reinstalling it seems to allow it to work. If both IE and Firefox are installed, both must be removed and IE 8 installed first followed by Firefox seems to allow IE 8 to work. Firefox - The 'Use TLS 1.0' must be selected but it still doesn't always seem to work. Sometimes (I can't tell you the characteristics), the customer must go through the remove and re-install process described above. I believe the remove and re-install process is likely setting some value to the one that allows the browser to work. Does anyone have any idea what that setting would be to avoid all this removing/re-installing? Thanks, Carl
Re: SSL with IBM JVM
Thanks for your reply. Interesting. Your suggestion is to front Tomcat with Apache and let Apache deal with SSL. I have tried to let Tomcat do everything, including serving html pages. I am going to try Chuck's idea of Sun's JVM 1.6.0_7 first. Thanks, Carl - Original Message - From: Mikolaj Rydzewski m...@ceti.pl To: Tomcat Users List users@tomcat.apache.org Sent: Monday, March 08, 2010 8:26 AM Subject: Re: SSL with IBM JVM Carl wrote: Switched to the latest IBM JVM last Friday. We run https for all applications as we deal mostly with children's data. About 10% of the users experienced problems with accessing the site. Both IE and Firefox caused problems. Switching them to http eliminated the problem but we can not run that way on a regular basis. Have you considered to use Apache httpd to maintain https traffic? You can use AJP or http tomcat connectors then. -- Mikolaj Rydzewski m...@ceti.pl - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: SSL with IBM JVM
Chuck, I have downloaded that JVM and brought it up on a spare server. We'll find out in the next week if it will work. Thanks, Carl - Original Message - From: Caldarale, Charles R chuck.caldar...@unisys.com To: Tomcat Users List users@tomcat.apache.org Sent: Monday, March 08, 2010 8:10 AM Subject: RE: SSL with IBM JVM From: Carl [mailto:c...@etrak-plus.com] Subject: SSL with IBM JVM I had been using the Sun JVM 1.6.0_18. I have tried 1.6.0_16 (because Taylan said his system started a similar problem when he went from 1.6.0_16 to 1.6.0_18) but not 1.6.0_9 (as Chuck suggested.) Minor correction: there is no 1.6.0_9; 1.6.0_7 (aka 6u7) was the last release prior to the major changes introduced in 6u10. Might be easier switching to that than the IBM JVM. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: SSL with IBM JVM
Peter, Good questions. I am dumping each request to catalina.out. I see the expected sequence of requests in the log. The sequence is: - Create two frames (one for work which covers the entire visible screen and one for communicating with the server for printing, applets, etc... an applet is started in this second frame which has a width of 0.) - In the visible frame, it simply display a 'Loading your settings...' message. - In the invisible frame, it starts the applet process by copying down some jars... we never see the requests to copy the jars. - Once the jars are copied, we see a request for the actual login page... this never happens on the machines that fail to copy the jars. The fact that it fails to request the jars implies to me (remember, this is the edge of my knowledge) the handshake negotiation was not going well and never completed so the server couldn't figure out how to communicate with the workstation. We normally have about 80 users from organizations we serve. About 20% of these appear to fail. I could take them through the process described earlier. However, we have several thousand users who are customers of these organizations and we can't really get them to do the uninstall/reinstall, hope it will work process. Hope this clarifies the problem a little bit. Thanks, Carl - Original Message - From: Peter Crowther peter.crowt...@melandra.com To: Tomcat Users List users@tomcat.apache.org Sent: Monday, March 08, 2010 8:57 AM Subject: Re: SSL with IBM JVM On 8 March 2010 12:53, Carl c...@etrak-plus.com wrote: Switched to the latest IBM JVM last Friday. We run https for all applications as we deal mostly with children's data. About 10% of the users experienced problems with accessing the site. Both IE and Firefox caused problems. Switching them to http eliminated the problem but we can not run that way on a regular basis. Here is what we have done: IE - IE 6 simply doesn't work. If IE 8 is installed, removing it and reinstalling it seems to allow it to work. If both IE and Firefox are installed, both must be removed and IE 8 installed first followed by Firefox seems to allow IE 8 to work. Firefox - The 'Use TLS 1.0' must be selected but it still doesn't always seem to work. Sometimes (I can't tell you the characteristics), the customer must go through the remove and re-install process described above. I believe the remove and re-install process is likely setting some value to the one that allows the browser to work. Does anyone have any idea what that setting would be to avoid all this removing/re-installing? There are a number of possibilities. Could you give us any more detail on which particular doesn't work you're getting? Browser hangs? Blank pages returned? Certificate errors? Error messages from browsers or server? - Peter - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: SSL with IBM JVM
Pid, I would never hijack a thread. I started this thread from scratch by sending it to the Tomcat users group... is that not what I have done? Thanks, Carl - Original Message - From: Pid p...@pidster.com To: users@tomcat.apache.org Sent: Monday, March 08, 2010 10:41 AM Subject: Re: SSL with IBM JVM On 08/03/2010 12:53, Carl wrote: I am trying the IBM JVM as a possible cure for the problem 'Tomcat dies suddenly'. A quick recap: New Dell T110 Slackware 13 - 64 bit Tomcat 6.0.24 The problem: Tomcat would die suddenly without any entry in any log but would leave a core file that indicated the JVM exited with a seg fault. I had been using the Sun JVM 1.6.0_18. I have tried 1.6.0_16 (because Taylan said his system started a similar problem when he went from 1.6.0_16 to 1.6.0_18) but not 1.6.0_9 (as Chuck suggested.) Switched to the latest IBM JVM last Friday. We run https for all applications as we deal mostly with children's data. About 10% of the users experienced problems with accessing the site. Both IE and Firefox caused problems. Switching them to http eliminated the problem but we can not run that way on a regular basis. Here is what we have done: IE - IE 6 simply doesn't work. If IE 8 is installed, removing it and reinstalling it seems to allow it to work. If both IE and Firefox are installed, both must be removed and IE 8 installed first followed by Firefox seems to allow IE 8 to work. Firefox - The 'Use TLS 1.0' must be selected but it still doesn't always seem to work. Sometimes (I can't tell you the characteristics), the customer must go through the remove and re-install process described above. I believe the remove and re-install process is likely setting some value to the one that allows the browser to work. Does anyone have any idea what that setting would be to avoid all this removing/re-installing? Thanks, Carl Carl, Please don't hijack threads. When you want to send an email to the list, just editing the subject line and the body isn't enough, most email clients leave the headers, (which contain the thread information), intact. p - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat dies suddenly
Chuck, Thanks, I will give it a try after I complete the two experiments currently underway (the IBM JVM and running with strace from the command line.) Tried bringing the server running the IBM JVM into production this morning (tested it yesterday with different browsers... can't use IE 6 but all the others seemed to work) but couldn't as there is some problem with SSL. The indicators are that some people can access the application just fine but others (they appear to be mostly IE users, will be getting the IE versions later this morning) never get past the first page (I can see from the log what is requested)... the first page starts a series of processes including copying some signed jars down, if needed, starting an applet that serves as a conduit for moving information to printers, etc. The odd part is all of these worked fine yesterday when we were using the firewall to redirect traffic on a specific port to this server (yesterday, we were sending traffic on port 8084 to 443 on this server, this morning we were sending all 443 traffic to this server.) More testing this morning but all ideas are certainly welcome. Thanks, Carl - Original Message - From: Caldarale, Charles R chuck.caldar...@unisys.com To: Tomcat Users List users@tomcat.apache.org Sent: Tuesday, March 02, 2010 10:10 PM Subject: RE: Tomcat dies suddenly From: Carl [mailto:c...@etrak-plus.com] Subject: Re: Tomcat dies suddenly Still looking for the exorcist but we now know that an older version of the Sun JVM doesn't do the trick. Not necessarily. 6u10 was a major internal upgrade, so it might be interesting to try 6u7: http://java.sun.com/products/archive/j2se/6u7/index.html (There were no 6u8 or 6u9 releases.) - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat dies suddenly
the problem. Thanks, Carl - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat dies suddenly
George, Thanks for the thoughts. My first thought was that the problem was hardware related and that the reason I could not see the problem was that the memtest86 did not sufficiently stress the machine to change the temperature enough to cause a failure. Subsequently, I built up another server with entirely different architecture (AMD vs Intel, different memory, different disks, etc.) and it failed in exactly the same manner. I have added memory to this second server just to test that we were not running out of memory by some fluke but those tests failed in exactly the same manner. My conclusion is that the problem is not hardware but rather either the Sun JVM (the only one I have used) or an errant piece of native code somewhere. I have tried to find the errant code by reducing the machine to the absolute minimum required to run the applicxation and that also showed the same failure. Chris suggested using strace (which I have) but I inadvertently overwrote the file containing the failure (not one of my brighter moves.) Thanks, Carl - Original Message - From: George Sexton geor...@mhsoftware.com To: 'Tomcat Users List' users@tomcat.apache.org Sent: Tuesday, February 23, 2010 7:45 PM Subject: RE: Tomcat dies suddenly -Original Message- From: Carl [mailto:c...@etrak-plus.com] Sent: Tuesday, February 23, 2010 5:09 AM To: Tomcat Users List Subject: Re: Tomcat dies suddenly Just an update. After 8 1/2 days, on the newly built Slackware machine with the JRE in the Slackware distribution removed bebore installing the operating system and using the newest version of the mysql-connector, the system failed in exactly the same fashion as the previous attempts: ran beautifully right up to the point of failure and the failure was the JVM being stopped with a reported seg fault. Changed this server to the IBM JVM. Tested it locally (directly accessed the IP within the DMZ) and it worked great. Switched it to production early this morning (4:30AM before people start coming onto the system) and everything seemed good. Then, specific customers (the rest were able to come in just fine) starting getting 404's (we use only https, didn't have a Carl, Just out of curiosity, have you tried building out machines with DIFFERENT hardware. E.G. building out a server using an IBM or HP computer, rather than than the ones you already have. If I recall correctly, you started this thread out with SIG 11's. SIG 11's on Linux are quite often hardware problems. I know you've done memtest, but sometimes that's not enough. Here's a link to a problem I had: http://archive.lug.boulder.co.us/Week-of-Mon-20071210/035903.html To make a long story short, there was random disk corruption that was happening. When I stopped using the on-board controller and went to a PCI one, the computer would reboot itself under heavy load. Some of the static burn-in utilities can miss hardware defects because they don't actually stress the system. E.G. power, CPU, disks, etc. The problem was a specific rev of a specific motherboard. I think you need to step back, get a computer from a different manufacturer and test. You've tried different OS's, different JVMs, different everything, but different hardware. By your own admission, the app used run flawlessly on an older server. When you've eliminated everything else, the thing that remains, however unlikely must be the culprit. George Sexton MH Software, Inc. http://www.mhsoftware.com/ Voice: 303 438 9585 - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: jvm exits without trace
Taylan, I am the person who started the Tomcat dies suddenly thread which I still haven't resolved. I am curious about the pattern of failures you are experiencing because they may provide some clues to my problem. In my case, the system will run for 15 minutes to 10 days before failing (most of the time it is several days to a week.) It appears to die from a seg fault in the JVM (I am using Sun 1.6.0_18 but have tried previous versions)... you may be able to see the cause of the failure from the core file (the core files on my systems were in several directories so you may have to do a 'find' to locate them.) Load may be a factor but the failures generally come after the load has been heavy for a while. I am running a couple of applications and it seems the failures are more frequent when people are hitting the additional apps (the primary app is always used, the remaining apps are used sporatically.) How does this compare to what you are experiencing? Thanks, Carl - Original Message - From: Taylan Develioglu tdevelio...@ebuddy.com To: Tomcat Users List users@tomcat.apache.org; p...@pidster.com Sent: Wednesday, February 24, 2010 5:09 AM Subject: Re: jvm exits without trace The GC log shows plenty of heap space left in all the spaces. I purposely didn't bother replacing the variables because I figured they would not be relevant. But if you think they might provide clues they're as follows: JAVA_HEAP_SIZE=18432M JAVA_EDEN_SIZE=$(($(echo $JAVA_HEAP_SIZE|sed 's/M$\|G$//')/6))M JAVA_PERM_SIZE=128M JAVA_STCK_SIZE=128K EDEN_SIZE is 1/6th of total heap. And I said there was nothing in the system logs. But you get a couple of points for trying. On Wed, 2010-02-24 at 10:44 +0100, Pid wrote: On 24/02/2010 09:36, Taylan Develioglu wrote: I thought I'd add the connector definitions too, : Connector port=80 protocol=org.apache.coyote.http11.Http11AprProtocol compression=1024 keepAliveTimeout=6 maxKeepAliveRequests=-1 enableLookups=false redirectPort=443 maxThreads=150 pollerSize=32768 pollerThreadCount=4/ Connector port=443 protocol=org.apache.coyote.http11.Http11AprProtocol SSLEnabled=true enableLookups=false maxThreads=10 scheme=https secure=true SSLCertificateFile=/etc/ssl/private/something.crt SSLCertificateKeyFile=/etc/ssl/private/something.key SSLCACertificateFile=/etc/ssl/certs/ca.crt/ On Wed, 2010-02-24 at 10:23 +0100, Taylan Develioglu wrote: Hi, I have jvm's, running tomcat and our application, exiting mysteriously, and was wondering if anyone could give me some advice on how to debug this thing. There is nothing in catalina.out, nor our application logs, and no hotspot error file. GC log looks normal. No trace in system logs. I am left completely clueless :(, has anyone dealt with a problem like this before? Any help appreciated. - Tomcat 6.0.24 - TC native 1.1.18 - APR 1.3.9 - Sun JDK 6u18 - Debian Lenny, 2.6.31.10-amd64 2 servlets, one as ROOT. 2 HTTP connectors that use TCNative/APR. JAVA_OPTS ( ): -verbose:gc -Djava.awt.headless=true -Dsun.net.inetaddr.ttl=60 -Dfile.encoding=UTF-8 -Djava.io.tmpdir=$TMP_DIR -Djava.library.path=/usr/local/lib -Djava.endorsed.dirs=$CATALINA_BASE/endorsed -Dcatalina.base=$CATALINA_BASE -Dcatalina.home=$CATALINA_HOME -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.util.logging.config.file=$CATALINA_BASE/conf/logging.properties -XX:+PrintGCDetails -Xloggc:$CATALINA_BASE/logs/gc.log -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=70 -Xms$JAVA_HEAP_SIZE -Xmx$JAVA_HEAP_SIZE -XX:NewSize=$JAVA_EDEN_SIZE -XX:MaxNewSize=$JAVA_EDEN_SIZE -XX:PermSize=$JAVA_PERM_SIZE -XX:MaxPermSize=$JAVA_PERM_SIZE -Xss$JAVA_STCK_SIZE -XX:+UseLargePages There's no actual heap size settings in the above. But you get a couple of points for trying. Google Linux Out Of Memory killer or OOM Killer and then check the server logs carefully. (e.g. /var/log/messages) p - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h
Re: jvm exits without trace
Taylan, The failures we've seen are in anywhere between 8 hours to a week of runtime. The timing of the failures seems similar. We have also had failures with hotspot error files (hs_err) present, and the cause specified was indeed SIGSEGV indicating a page fault. I have never seen any hs_* files but have seen core files where strace showed the jvm stopped on a seg fault. We also use jdk 1.6.0_18, I'm downgrading the machines to 1.6.0_16 when the situation allows (during regular updates of the application, or a crash) to see if that helps. I have used jdk 1.6.0_17 and 1.6.0_18 with the same results... have not tried 1.6.0_16. Please post your results of this trial. Running tomcat on the foreground might show something, but then again I could be waiting for a month for it to happen. Yes, this has been part of my problem as anytime we change something, we have to wait a week for the server to fail. In one sense, I am fortunate that I have a little more flexibility than you. I have two servers (different hardware) but only need one in service at a time. Therefore, I always have one server I can test ideas on although I have never been able to develop a meaningful stress test, i.e., the only way I can test a change is to put it in production. Thanks, Carl - Original Message - From: Taylan Develioglu tdevelio...@ebuddy.com To: Tomcat Users List users@tomcat.apache.org Sent: Wednesday, February 24, 2010 8:31 AM Subject: Re: jvm exits without trace Hello Carl, The failures we've seen are in anywhere between 8 hours to a week of runtime. Most of them have (still) been running for almost a month without failure. There are ~100 machines. From the top of my head, I think we've had about 10+ failures now. We have also had failures with hotspot error files (hs_err) present, and the cause specified was indeed SIGSEGV indicating a page fault. But I don't know if the two are related. We also use jdk 1.6.0_18, I'm downgrading the machines to 1.6.0_16 when the situation allows (during regular updates of the application, or a crash) to see if that helps. It might be useful to note that the failures happen with tomcat 6.0.20 as well as 6.0.24. As far as load concerns, I haven't had a failure on an idle machines. The machines are well loaded, but only at a fraction limit in regards to load and cpu utilization. Most memory is commited to tomcat, where a 24G machine would have 18G allocated to heap, 128M to permgen and some unspecified amount would get used by jni for apr. About 4G remains free after calculating taking into account the jvm itsself. A 16G machine would have 12G allocated to the heap. Besides the fact that our apps heavily use nio and mina I wouldn't say there's anything else noteworthy. There can be anywhere up to 1 concurrents on one machine. I had searched for coredumps, but no luck. Running tomcat on the foreground might show something, but then again I could be waiting for a month for it to happen. On Wed, 2010-02-24 at 12:42 +0100, Carl wrote: Taylan, I am the person who started the Tomcat dies suddenly thread which I still haven't resolved. I am curious about the pattern of failures you are experiencing because they may provide some clues to my problem. In my case, the system will run for 15 minutes to 10 days before failing (most of the time it is several days to a week.) It appears to die from a seg fault in the JVM (I am using Sun 1.6.0_18 but have tried previous versions)... you may be able to see the cause of the failure from the core file (the core files on my systems were in several directories so you may have to do a 'find' to locate them.) Load may be a factor but the failures generally come after the load has been heavy for a while. I am running a couple of applications and it seems the failures are more frequent when people are hitting the additional apps (the primary app is always used, the remaining apps are used sporatically.) How does this compare to what you are experiencing? Thanks, Carl - Original Message - From: Taylan Develioglu tdevelio...@ebuddy.com To: Tomcat Users List users@tomcat.apache.org; p...@pidster.com Sent: Wednesday, February 24, 2010 5:09 AM Subject: Re: jvm exits without trace The GC log shows plenty of heap space left in all the spaces. I purposely didn't bother replacing the variables because I figured they would not be relevant. But if you think they might provide clues they're as follows: JAVA_HEAP_SIZE=18432M JAVA_EDEN_SIZE=$(($(echo $JAVA_HEAP_SIZE|sed 's/M$\|G$//')/6))M JAVA_PERM_SIZE=128M JAVA_STCK_SIZE=128K EDEN_SIZE is 1/6th of total heap. And I said there was nothing in the system logs. But you get a couple of points for trying. On Wed, 2010-02-24 at 10:44 +0100, Pid wrote: On 24/02/2010 09:36, Taylan Develioglu wrote: I thought I'd add the connector definitions too, : Connector port=80 protocol=org.apache.coyote.http11
Re: Tomcat dies suddenly
Chuck, The first core that I saw was in $CATALINA_HOME/bin as expected. The next crash produced one at root. I thought this might have been because there is a cron process that runs a shell script in the /usr/local/tomcat_wars directory that restarts Tomcat (using shutdown.sh and startup.sh in the $CATALINA_HOME/bin directory) at 1:00AM so I just never paid it much attention. Red herrings, green herrings, blue herrings... who knows anymore? Thanks, Carl - Original Message - From: Caldarale, Charles R chuck.caldar...@unisys.com To: Tomcat Users List users@tomcat.apache.org Sent: Wednesday, February 24, 2010 10:00 AM Subject: RE: Tomcat dies suddenly From: Carl [mailto:c...@etrak-plus.com] Subject: RE: Tomcat dies suddenly (I switched this comment to Carl's original thread, since that's where it applies.) the core files on my systems were in several directories I wonder if the above is a clue to what's going on. I've always seen core files written to the current directory of the dying process, never scattered around arbitrarily. Tomcat's current directory should be $CATALINA_HOME/bin; if one of the webapps is changing the current directory, could that be contributing to the failure by confusing other code or the JVM? (Or is this just another of the many red herrings?) - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: [OT] Tomcat dies suddenly
Chris, As I said in an email a minute ago, I use a cron process to stop Tomcat, copy in new war's and restart Tomcat at 1:00AM. The script is pretty simple: JAVA_HOME=/usr/local/java; export JAVA_HOME cd /usr/local/tomcat/bin ./shutdown.sh sleep 60 rm -rf /usr/local/tomcat/webapps/livonia rm -rf /usr/local/tomcat/webapps/default rm -rf /usr/local/tomcat/webapps/paragon cp /usr/local/tomcat_wars/etrak-plus.war /usr/local/tomcat/webapps/livonia.war cp /usr/local/tomcat_wars/etrak-plus.war /usr/local/tomcat/webapps/default.war cp /usr/local/tomcat_wars/etrak-plus.war /usr/local/tomcat/webapps/paragon.war cp /usr/local/tomcat_wars/etrak-plus.war /usr/local/tomcat/webapps/test.war sleep 30 strace -o /usr/local/tomcat/logs/trace_log.txt /usr/local/tomcat/bin/startup.sh exit I am reworking the strace part to rotate the logs. Thanks, Carl - Original Message - From: Christopher Schultz ch...@christopherschultz.net To: Tomcat Users List users@tomcat.apache.org Sent: Wednesday, February 24, 2010 10:15 AM Subject: Re: [OT] Tomcat dies suddenly -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Chuck, On 2/24/2010 10:00 AM, Caldarale, Charles R wrote: From: Carl [mailto:c...@etrak-plus.com] Subject: RE: Tomcat dies suddenly the core files on my systems were in several directories I wonder if the above is a clue to what's going on. I've always seen core files written to the current directory of the dying process, never scattered around arbitrarily. Tomcat's current directory should be $CATALINA_HOME/bin; Mine never is, but I run using catalina.sh, which doesn't execute a 'cd' anywhere. Maybe this happens when running as a Windows service or something, but I don't see this behavior when running on *NIX. if one of the webapps is changing the current directory, could that be contributing to the failure by confusing other code or the JVM? (Or is this just another of the many red herrings?) I suppose that's possible. Carl, are the core files all from the same run? If so, something is very very wrong. Otherwise, maybe the reason why you couldn't find any traces of your crash was that your CWD was changing from what you expected it to be to some random subdirectory. shrug - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkuFQnoACgkQ9CaO5/Lv0PBZYACgjd19BIr41ucOAenBQVE7//S6 ltoAnRyndQdsC8h70kshy1p7bkVgi6nY =+ndQ -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat dies suddenly
Just an update. After 8 1/2 days, on the newly built Slackware machine with the JRE in the Slackware distribution removed bebore installing the operating system and using the newest version of the mysql-connector, the system failed in exactly the same fashion as the previous attempts: ran beautifully right up to the point of failure and the failure was the JVM being stopped with a reported seg fault. Changed this server to the IBM JVM. Tested it locally (directly accessed the IP within the DMZ) and it worked great. Switched it to production early this morning (4:30AM before people start coming onto the system) and everything seemed good. Then, specific customers (the rest were able to come in just fine) starting getting 404's (we use only https, didn't have a chance to see if they could come in as http) on their first access. I switched to a backup server with the old configuration and everyone seems OK for now. Remember, three servers: A - Slackware 13 - 64 bit. Latest IBM JVM. Tomcat 6.0.24. B - Slackware 13 - 64 bit. Sun JVM 1.6.0_18. Tomcat 6.0.24. C - Slackware 12 - 32 bit. Sun JVM 1.5.0_01. Tomcat 5.5.23 A is the server I have been experimenting with. B is the server I am currently running on and expect it to fail (because it always has.) C is the original server that has run successfully for several years. One more event that may or may not be related. I have not touched the original server (C) except to roll new war's out (almost nightly... minor bug fixes.) This morning, it started producing these messages every 10 seconds: Feb 23, 2010 5:50:01 AM org.apache.catalina.loader.WebappClassLoader modified INFO: Additional JARs have been added Feb 23, 2010 5:50:01 AM org.apache.catalina.core.StandardContext reload INFO: Reloading this Context has started Chuck, seems like it is time to break out the exorcism tools. Thanks, Carl - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat dies suddenly
This was an 'oh, crap' moment. Andre was correct: I had commented a shutdown out of the script I used to deploy (each morning at 1:00AM.) So, apparently, it was just merrily reploying and redeploying, or something. All better now (with the proper deploy script.) Thanks to everyone and have a good day. Carl - Original Message - From: Caldarale, Charles R chuck.caldar...@unisys.com To: Tomcat Users List users@tomcat.apache.org Sent: Tuesday, February 23, 2010 8:54 AM Subject: RE: Tomcat dies suddenly From: Pid [mailto:p...@pidster.com] Subject: Re: Tomcat dies suddenly Check the server time hasn't drifted. Sometimes this can happen if the files have timestamps that are out. Also check the timestamps on the .war files - they may be in the future, causing continuous redeployment. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat dies suddenly
Chris, There was no core dump or hs_* file. The strace output looks like it was overwritten this morning at 1:00AM, crap, double crap. What's the consensus on moving to the IBM JVM or rerunning this test (Sun JVM) to failure to get a good strace output? I screwed up... sorry. Thanks, Carl - Original Message - From: Christopher Schultz ch...@christopherschultz.net To: Tomcat Users List users@tomcat.apache.org Sent: Tuesday, February 23, 2010 1:13 PM Subject: Re: Tomcat dies suddenly -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Carl, On 2/23/2010 7:08 AM, Carl wrote: Just an update. After 8 1/2 days, on the newly built Slackware machine with the JRE in the Slackware distribution removed bebore installing the operating system and using the newest version of the mysql-connector, the system failed in exactly the same fashion as the previous attempts: ran beautifully right up to the point of failure and the failure was the JVM being stopped with a reported seg fault. ...and?! What does strace say? Or, the hs_* file that should have been dumped? Or your core file or whatever? - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkuEGrUACgkQ9CaO5/Lv0PCwBACfb7RjY+gFnKIe3I0WWuwZvywo aIAAn1NdVr0Pp8HGsK74arolpbKWrwrO =F403 -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat dies suddenly
Just an update. A quick refresher. Server A has been rebuilt with eliminating the JRE before Slackware is installed. The jre from Slackware was allowed to be installed and was then removed using the package tool. The applications on both servers have been upgraded to mysql-connector version 5.1.11 from 3.1.12. Have been running on server A for four days witout incident (before I start celebrating, I remind myself that, at times, the system ran for 10 days before failing.) Yesterday, there was a moderate load and people were accessing several of the applications... I plan to force that today as that seemed to provoke the failure in the past. However, as Chris pointed out, I won't know if it was the change in the Slackware install or the change in the mysql-connector. If we go another week without failure, I plan to switch to server B to see if I can make it fail. Sure would be nice to know that we actually cured the problem. Andre - we don't use OpenSSL. Chris - I am running inside of strace. If we do get a failure, I am hping to get at least some information. You noted that it may take several cycles, I agree. Tony - I haven't tried brining up a separate application to see how it bevaed under load. I have used JMeter to stress the server (don't remember which one) repeatedly hitting a small corner of the application... the server behaved perfectly, even with out of memory conditions. Mark - I have not yet tried the IBM JVM because I am waiting to see the results of the current tests. I am trying to go slowly enough to know that I have actually solved the problem not just befuddled it. And, with a 10 day cycle time, testing options can take a long time. In my area (Charleston, SC) there are people, mainly ministers, that claim they can cure health issues by laying their hands on a person and saying a few words. I was thinking maybe Andre could do the same for my servers, you know, with his telepathy thingie and all. Thanks, Carl - Original Message - From: Christopher Schultz ch...@christopherschultz.net To: Tomcat Users List users@tomcat.apache.org Sent: Tuesday, February 16, 2010 4:55 PM Subject: Re: Tomcat dies suddenly -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 André, On 2/16/2010 4:41 PM, André Warnier wrote: So what now ? I leave this list for 3 days to go earn my keep, and when I come back this silly crash issue is /still/ not resolved ? Well, we've been waiting for you to bring your expertise to bear on this issue, André :) still on my trip about the SegFaults caused by some native non-kosher executable module.. I'd be surprised if this was the problem: Carl is installing Sun's JRE, which should have everything it needs. He says he's not playing around with any native code, so I wouldn't expect anything to be incompatible. I'm not saying it's impossible... just that I'd be surprised if it's the case. I'd think you'd get something other than a SIGSEGV anyway... attempting to load an incompatible library ought to get you some other error... though I'm not sure which. You could probably try to force that to happen with something like this (in Java, of course): System.loadLibrary(my-bad-arch.so); ...and see what happens. I don't have a library of the wrong architecture laying around right now. I tried to load some random file, and I got an UnsatisfiedLinkError, but that's at the Java level. I have no idea what would happen if the native X11 graphics library was being loaded by the JVM and wasn't compatible... Appeal to the experts : considering the lenghtily expounded symptoms, would it be possible that one obscure corner of the SSL code or data is being used only occasionally, and that such occasional usage could cause the symptoms here observed ? It's possible. If SSL is in use, consider downgrading to the pure Java connectors (i.e. no APR... enjoy re-configuring Tomcat to use the different-style SSL configuration) to see if that clears things up. strace would indicate that this was the problem if you had multiple runs: the openssl libraries would always be in the mix of the stack trace. - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkt7FDsACgkQ9CaO5/Lv0PC1cgCfRf9qHNkMJEs54I/CCAxuzBc8 zroAnRLGS53mv6V/T1sxur1vEekbd3Ym =XaqE -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat dies suddenly
Chris, Yup, that was my plan. Thanks, Carl - Original Message - From: Christopher Schultz ch...@christopherschultz.net To: Tomcat Users List users@tomcat.apache.org Sent: Wednesday, February 17, 2010 10:02 AM Subject: Re: Tomcat dies suddenly -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Carl, On 2/17/2010 8:14 AM, Carl wrote: A quick refresher. Server A has been rebuilt with eliminating the JRE before Slackware is installed. The jre from Slackware was allowed to be installed and was then removed using the package tool. The applications on both servers have been upgraded to mysql-connector version 5.1.11 from 3.1.12. Okay. We'll see after a few days if you're good to go. Downgrading the MySQL Connector/J driver should be as simple as replacing the JAR file and bouncing Tomcat, so that's an easy test in and of itself. - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkt8BOsACgkQ9CaO5/Lv0PCouQCeOZ0TWGZSMnoMUHvaI6bQdR+p RwsAnjDokDvU1csXUnyi1QIkkbJKVanb =NNNn -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: [OT] Tomcat dies suddenly
Jorge, If the problem was easy, there wouldn't be the number of threads. I am running the standard JVM from Sun, in fact, have tried two versions. The boxes are not the problem because the problem appears on two very different boxes. Thanks, Carl - Original Message - From: Jorge Medina cerebrotecnolog...@gmail.com To: Tomcat Users List users@tomcat.apache.org Sent: Sunday, February 14, 2010 12:44 AM Subject: Re: [OT] Tomcat dies suddenly There have been 144 messages on this thread...and you have spent already months trying to solve the problem...I think it will be more cost effective to replace the boxes, run a standard JVM from Sun..and close this thread! On Sat, Feb 13, 2010 at 6:11 PM, Caldarale, Charles R chuck.caldar...@unisys.com wrote: From: André Warnier [mailto:a...@ice-sa.com] Subject: Re: [OT] Tomcat dies suddenly Maybe we should also investigate if the SegFaults are simultaneous with anyone specific entering the room where the servers are. Ah yes, the old nylon underwear problem... Or the pizza with plutonium toppings. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: [OT] Tomcat dies suddenly
Anthony, Very certain it is not the hardware, am running Sun JVM, could be the OS. May be my next step (CentOS.) Thanks, Carl - Original Message - From: anthonyvie...@gmail.com To: Tomcat Users List users@tomcat.apache.org Sent: Sunday, February 14, 2010 1:01 AM Subject: Re: [OT] Tomcat dies suddenly CentOS, Sun JVM, IBM Hardware = 100% Uptime On Sat, Feb 13, 2010 at 9:44 PM, Jorge Medina cerebrotecnolog...@gmail.comwrote: There have been 144 messages on this thread...and you have spent already months trying to solve the problem...I think it will be more cost effective to replace the boxes, run a standard JVM from Sun..and close this thread! On Sat, Feb 13, 2010 at 6:11 PM, Caldarale, Charles R chuck.caldar...@unisys.com wrote: From: André Warnier [mailto:a...@ice-sa.com] Subject: Re: [OT] Tomcat dies suddenly Maybe we should also investigate if the SegFaults are simultaneous with anyone specific entering the room where the servers are. Ah yes, the old nylon underwear problem... Or the pizza with plutonium toppings. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat dies suddenly
Peter, That's what I thought. Thanks, Carl - Original Message - From: Peter Crowther peter.crowt...@melandra.com To: Tomcat Users List users@tomcat.apache.org Sent: Sunday, February 14, 2010 5:29 AM Subject: Re: Tomcat dies suddenly On 13 February 2010 15:29, Caldarale, Charles R chuck.caldar...@unisys.comwrote: Does memtest86+ fire up enough threads to heat up all the cores? No. It's a pure memory tester, and it's single-threaded so that it always knows what's adjacent to each bit. - Peter - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat dies suddenly
Mark, Thanks for the advice. I started down the 32 bit path but had difficulty with missing libraries on 64 bit Slackware. Brought up Slackware 32 bit but realized that was going back to the restrictive memory problems so stopped that path. I had thought about using a different JVM but wasn't certain which ones were truely different and not just a fork of Sun's JVM. You have clarified that for me. Thanks, Carl - Original Message - From: Mark Thomas ma...@apache.org To: Tomcat Users List users@tomcat.apache.org Sent: Sunday, February 14, 2010 6:58 AM Subject: Re: Tomcat dies suddenly On 13/02/2010 22:23, Carl wrote: 5) not quite sure of this anymore, but it seems to happen also on different JVMs, which would tend to rule out a problem with a particular JVM port. No, I have only used Sun's 64 bit. Started with 1.6.0_17 and am now using 1.6.0_18. That is enough of a common factor in all of the failures that it worth looking to see if changing it makes a difference. I'd see if you get the same results with other JVMs. The ones I'd try are: - Sun 32-bit 1.6.0_18 (see if 32/64 makes a difference) - IBM 64-bit 1.6 SR7 (see if IBM/Sun makes a difference) - Sun 64-bit 1.6.0_07 (see if an older JVM from before the large(ish) changes in 1.6.0_10 makes a difference) Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: [OT] Tomcat dies suddenly
George, Been there. In Brookfield (CT), under certain circumstances, when someone would use the copier (not on the same circuit), the Novell server would go down. Here, we have all new wiring directly from the fuse box with really good UPS's in between. Thanks, Carl - Original Message - From: George Sexton geor...@mhsoftware.com To: 'Tomcat Users List' users@tomcat.apache.org Sent: Sunday, February 14, 2010 11:58 AM Subject: RE: [OT] Tomcat dies suddenly -Original Message- From: André Warnier [mailto:a...@ice-sa.com] Sent: Saturday, February 13, 2010 3:28 PM To: Tomcat Users List Subject: Re: [OT] Tomcat dies suddenly Caldarale, Charles R wrote: Since we must have by now exhausted all the normal causes of such errors, maybe we should recommend a) a visual inspection of the systems, to see if there are any pinsize holes, or paint flaking off or so b) the installation of a surveilance camera, to check if the SegFaults are synchronous with any visible phenomenon (sparks, Cerenkow radiation, etc.) c) moving the systems to the basement ? There's a story in a book I once read where a computer system crashed every morning around the same time. No one could figure it out. Finally, the head of IS goes down to the computer room at the expected time. In walks a maintenance man who comes in, opens the cabinet for the computer, and plugs a floor polisher into a spare outlet in the cabinet. When the maintenance man activates the polisher, boom, the system crashed. When asked by his boss what the problem was, he told him it was a buffer problem. http://www.amazon.com/Devouring-Fungus-Jennings-Karla/dp/0393307328 I used to have a Novell server that would mysteriously reboot every few days. In my case, the server was on the same circuit as a laser printer, and both were plugged into a Haworth cubicle outlet. Periodically, load was too much and it would causes a server reboot. We brought in another circuit NOT on the inadequate cubicle wiring system, and the problem went away. George Sexton MH Software, Inc. http://www.mhsoftware.com/ Voice: 303 438 9585 - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat dies suddenly
Chris, I upgraded to 5.1.11 but can go back to 3.1.12 in about 15 minutes. Have been running all day on 5.1.11... no burps so far but I haven't been pushing it. Tomoroow, I will see if I can break it. Thanks, Carl - Original Message - From: Christopher Schultz ch...@christopherschultz.net To: Tomcat Users List users@tomcat.apache.org Sent: Sunday, February 14, 2010 4:15 PM Subject: Re: Tomcat dies suddenly -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Carl, On 2/13/2010 5:35 PM, Carl wrote: I am using the mysql-connector/j version 3.1.12. Interesting because the latest driver is version 5.1.11. Is this worth a shot or is it likely to just miuddy the waters? We typically have less than 20 open connections. To my knowledge, MySQL's Connector/J has never had a native driver: they've all been Type 4 drivers, so native code still isn't a problem. I'd upgrade to a new version of the driver eventually, but probably not until you get everything else stable. - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkt4Z+UACgkQ9CaO5/Lv0PCaCgCaA5OMR96UMLX6BuIxQ+nGvpDV edIAniumoLaxFYzQe/LUflUFX7jDCH3D =Dt6R -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat dies suddenly
Chris, I agree it will be several crashes (maybe never) but strace is a low risk, low investment so I am running with it. I am also using your suggestion to trap the exit code although I believe we already know what it will be. The production server that has held up all day is the T110 which I did a fresh install and elimin ated the packaged jre prior to installing the operating system. My hypothesis is that the package removal process may have left something hanging around. Thanks for the suggestions. Carl - Original Message - From: Christopher Schultz ch...@christopherschultz.net To: Tomcat Users List users@tomcat.apache.org Sent: Sunday, February 14, 2010 4:19 PM Subject: Re: Tomcat dies suddenly -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Carl, On 2/13/2010 4:04 PM, Carl wrote: I will start the newly rebuilt server with strace tomorrow morning before anyone comes on. Hopefully, strace will yield some useful information. You'll likely have to run your server repeatedly under strace in order to gather any meaningful data. Honestly, it would be a miracle if every crash appeared to be for the same function call. Good luck, - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkt4aM8ACgkQ9CaO5/Lv0PC7NwCeJz4rJgZ9vvgaIyNWJ39K5Zr2 XN4An3qqqONIrJNlnrUoZ00rr+uiAQ93 =GlP1 -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: [OT] Tomcat dies suddenly
Chris, You are evil. Thanks, Carl - Original Message - From: Christopher Schultz ch...@christopherschultz.net To: Tomcat Users List users@tomcat.apache.org Sent: Sunday, February 14, 2010 4:21 PM Subject: Re: [OT] Tomcat dies suddenly -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Anthony, On 2/14/2010 1:01 AM, anthonyvie...@gmail.com wrote: CentOS, Sun JVM, IBM Hardware = 100% Uptime 100%, eh? Care to make a wager? - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkt4aXQACgkQ9CaO5/Lv0PDdqgCdFW5C4rWO4XwCI9/EK9APlk+K CXIAn3akAol+pBE12guiAz+krXytzV1T =w3u5 -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat dies suddenly
Chris and Andre, Andre's note that it was always code that was not meant for the platform triggered a thought that it might be remnants of the jre Slackware includes in their distribution. Let me explain. I have been installing Slackware by just saying 'load everything'. Then, I would remove the jre 'package' using the package manager. My thought was what if the package manager is not removing everything? So, I am rebuilding one of the servers eliminating unwanted packages before they are installed (take less than 30 minutes... not certain how I can get a 10 minute test to see if I accomplished anything.) I agree with Chris that the only definitive way of finding the problem is to get a stack trace. It seems to me we have two stack traces that we need to know about: 1) the jvm stack trace and 2) the java stack trace. Running gdb against the core dump only tells me the problem was in the jvm because there is no debugging info in the jvm. So, the only way to get the details would seem to be to build the jvm from source (I have downloaded the source but haven't built the jvm yet.) I don't know how to force a java stack dump at point of failure, not even certain it is possible because it would seem the the failure in the C code in the jvm would mean the jvm would stop before it could give a stack trace. Understand that this is my best guess and that this area is removed from my usual mundane Java application development. If anyone has suggestions, I am open to them because I know I know very little. Thanks, Carl - Original Message - From: Christopher Schultz ch...@christopherschultz.net To: Tomcat Users List users@tomcat.apache.org Sent: Saturday, February 13, 2010 9:46 AM Subject: Re: Tomcat dies suddenly -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 André, On 2/12/2010 7:29 PM, André Warnier wrote: I would just like to mention that in 90% of the cases where I have seen a Seg Fault, it was due to the attempted execution of a piece of binary code not meant for the current platform. (It's been a while since I've seen one though.) In a Java context, for me it's always been either misbehaving native code (/not/ from Sun... this would be application code), or bad hardware. Maybe another run through memtest86+ would be a good idea. I'd love to see a stack trace from a few crashes, though. - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkt2u18ACgkQ9CaO5/Lv0PBGsQCgvhnXtIby1uP47o3BmjN8Hlyh USAAn1P/xLbv3tDhsTto6lWXDfwd4lM7 =xovn -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat dies suddenly
Chuck, I don't know. Memtest86 states that it has 'support for execution with multiple CPUs' but I recall that process froze when I tried it and the suggestion (from memtest86) was to use a different option. I will revisit this after I have tried rebuilding the server sans jre and building the jvm from source (so I can use it with gdb.) Thanks, Carl - Original Message - From: Caldarale, Charles R chuck.caldar...@unisys.com To: Tomcat Users List users@tomcat.apache.org Sent: Saturday, February 13, 2010 10:29 AM Subject: RE: Tomcat dies suddenly From: Christopher Schultz [mailto:ch...@christopherschultz.net] Subject: Re: Tomcat dies suddenly Maybe another run through memtest86+ would be a good idea. Does memtest86+ fire up enough threads to heat up all the cores? - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat dies suddenly
Andre,
I tried this and 1) I am now permamently cross eyed and 2) didn't see
anything that was out of place or looked like a binary that should not be
there.
Thanks,
Carl
- Original Message -
From: André Warnier a...@ice-sa.com
To: Tomcat Users List users@tomcat.apache.org
Sent: Saturday, February 13, 2010 11:49 AM
Subject: Re: Tomcat dies suddenly
Just a quick way to check if a rogue binary hasn't crept into some
libraries directory :
find . -type f -print -exec file {} \; | more
That should give the same file type most of the time, except the rogue
module.
(the -print may be superfluous)
-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat dies suddenly
Tsirkin, I tried part of that. I brought up a server with openSuse (64 bit), the latest Sun Java and the latest Tomcat. Failed in about 15 minutes with the same indicators (no tracks in any log, didn't know to look for a core file at that time.) Could try this again and check for the core file to see if the failure is the same. I agree building from source and debugging is a very hard road. Have been trying to find a solution for almost three months and everything I have tried has failed. There appear to be only two moving parts: the operating system and the jvm (we now know the failure is when the jvm seg faults.) Maybe I should try a different jvm but I have always believed Sun's was most likely the most stable and bug free. Another option is to create a separate Tomcat for each application. This would require reworking and rethinking the applications with no guarantee of success anyway. It would seem that there is something wrong in my setup because I can't believe every 64 bit Slackware/Tomcat has failed as we would likely see that on this list. I am certainly open to any suggestion and I appreciate your thoughts. Thanks, Carl - Original Message - From: Tsirkin Evgeny tsir...@gmail.com To: Tomcat Users List users@tomcat.apache.org Sent: Saturday, February 13, 2010 1:24 PM Subject: Re: Tomcat dies suddenly Carl , At this point it is probably would be much simpler for you to just move away from Slackware . Building jvm from source ,debugging with strace - this is a very hard path . And once you find the bug - there is nothing that you can do with it. You are not going to fix jvm/libc bugs ,right? You could report it and wait for new release . Probably your best bet would be use another distro and download Sun's jvm from thier site. Evgeny So, the only way to get the details would seem to be to build the jvm from source (I have downloaded the source but haven't built the jvm yet.) I don't know how to force a java stack dump at point of failure, not even certain it is possible because it would seem the the failure in the C code in the jvm would mean the jvm would stop before it could give a stack trace. Understand that this is my best guess and that this area is removed from my usual mundane Java application development. If anyone has suggestions, I am open to them because I know I know very little. Thanks, Carl - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat dies suddenly
The oomParachute does not seem a likely candidate for solving the issue because 1) we have never seem the memory (from JConsole or VisualJVM) fill the heap or approach the max memory in the machine (never uses swap) or come close to blowing the permGem memory. More and more it does not seem like a memory problem. Thanks, Carl - Original Message - From: Christopher Schultz ch...@christopherschultz.net To: Tomcat Users List users@tomcat.apache.org Sent: Friday, February 12, 2010 5:28 PM Subject: Re: Tomcat dies suddenly -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 André, On 2/12/2010 3:34 PM, André Warnier wrote: Carl wrote: Andre, Thanks for the response. I have read almost all of your posts and realy enjoy the way to take problems apart. Keep on thinking. I'm not quite sure how to take the above answer.. So, just in case, and maybe to my own ultimate embarassment, I want to indicate that I was serious. I seem to remember cases where an application at the point of dying, would have very much liked to log a last desperate message to indicate the situation, but did not even have the resources left to be able to do so. Are you talking about this? http://tomcat.apache.org/tomcat-6.0-doc/config/http.html (search for oomParachute) - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkt11hkACgkQ9CaO5/Lv0PA40gCfaqBCAz8wq2k6W3SH3gKIlF7q xMQAnjnynEOuXlosG/v2jWHVx5akaQWo =ODoh -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat dies suddenly
Chris, This is the only thing I see from gdb: r...@tomcat_liv:/# gdb -c core GNU gdb 6.8 Copyright (C) 2008 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type show copying and show warranty for details. This GDB was configured as x86_64-slackware-linux. (no debugging symbols found) Core was generated by `/usr/local/java/bin/java -Djava.util.logging.config.file=/usr/local/tomcat/conf'. Program terminated with signal 11, Segmentation fault. [New process 3824] [New process 4182] [New process 3811] [New process 3823] [New process 3825] [New process 4325] [New process 3849] [New process 3364] [New process 3850] [New process 3393] [New process 3851] [New process 3395] [New process 3852] [New process 3399] [New process 3401] [New process 3853] [New process 3406] [New process 3859] [New process 3860] [New process 3861] [New process 3862] [New process 3863] [New process 3410] [New process 3864] [New process 3880] [New process 3416] [New process 3939] [New process 3940] [New process 3775] [New process 3986] [New process 3780] [New process 3987] [New process 3388] [New process 4291] [New process 3387] [New process 3403] [New process 3383] [New process 3389] [New process 3396] [New process 3398] [New process 3407] [New process 3408] [New process 3409] [New process 3411] [New process 3412] [New process 3413] [New process 3414] [New process 3415] [New process 3776] [New process 3782] [New process 3818] [New process 3820] #0 0x7fe01f9d359d in ?? () I have thought the reason I am seeing nothing beyond the JVM is that the JVM has no debugging symbols. Did I miss something? Thanks, Carl - Original Message - From: Christopher Schultz ch...@christopherschultz.net To: Tomcat Users List users@tomcat.apache.org Sent: Friday, February 12, 2010 5:27 PM Subject: Re: Tomcat dies suddenly -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Carl, On 2/12/2010 4:59 PM, Carl wrote: Darn, I thought we were onto something here but, as you suspected, the line contains a lot of parameters and was truncated. So, now, I think we know the JVM is seg faulting, we just don't know where or why. I'm guessing we have to somehow get a stack trace at the point of failure... any idea how I can get one? Or, is there a better way? I believe you can do roughly this with gdb (from memory): $ gdb core-file gdb) where (boom: your stack trace goes here) - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkt11bwACgkQ9CaO5/Lv0PBUCgCfbR2f2IXwFq7ile8biFNjsXOH opEAnj7gYfb2jfQDtwIcl5atUpyYG8au =im6P -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat dies suddenly
Chris, I find it hard to believe two brand new machines with different processors, etc. would have a hardware problem that showed itself in exactly the same way. Further, I have run memTest86 for 30 hours on one of the servers and it showed nothing (although, as Chuck pointed out, the test may not have handled the cores correctly or may not have changed the temperature sufficiently to cause the problem we are seeing.) I have not found a mem test specifically for 64 bit processors. Thanks, Carl - Original Message - From: Christopher Schultz ch...@christopherschultz.net To: Tomcat Users List users@tomcat.apache.org Sent: Friday, February 12, 2010 5:26 PM Subject: Re: Tomcat dies suddenly -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Carl, On 2/12/2010 4:44 PM, Carl wrote: Now, this is embarrassing: I just checked the other server and it also has a core file with the date and time of the last failure in the tomcat/bin directory. And, it shows a seg fault at exactly the same code. This might be a winner... we certainly know it killed the java process. So, this now, to me, narrows this down to two possibilities: 1. A JVM bug 2. A hardware problem That is, if you really aren't running any native code. But, if you were, it would be showing up in the code dump, right?! - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkt11YMACgkQ9CaO5/Lv0PC/ZwCcCdB3k1ARO5uhxneWilt3wkox n/4AniRJb/t3Xd9FgKQecXHN7UyFP5RQ =s/EK -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat dies suddenly
Donn,
It looks like the total files opened are less than 1,000 and the ulimit is
set to 4,096 (this was increased as a way to check if ulimit was a
problem... did not change the behavior of the system.)
We use jdbc with the commons pooling process. We follow the number of open
connections very closely (logging to catalina.out) because we have had
connection leaks (still have a small one) in the past.
We do not use LDAP.
Thanks,
Carl
- Original Message -
From: Donn Aiken donn.ai...@gmail.com
To: Tomcat Users List users@tomcat.apache.org
Sent: Friday, February 12, 2010 4:00 PM
Subject: Re: Tomcat dies suddenly
Carl -
I did have something like this happen to me - not with Tomcat but with
another JEE container. The container would run for a while, without
incident, then suddenly simply die, with nothing in any log, and not on any
apparent time schedule.
We had some code that was manipulating LDAP that had a leak in it. For each
leaked connection, we had an open file descriptor that never went away,
until the process went away. If memory serves, we finally found it by
looking at entries in /proc/{pid of jvm}/fd, doing a bunch of find . | wc
and watching that over time.
I hope this is of some help. Good luck.
Donn
On Fri, Feb 12, 2010 at 3:46 PM, Carl c...@etrak-plus.com wrote:
Andre,
Take my comment as a compliment because that is the way it was meant...
you
have helped a lot of people on this least and I, for one, really
appreciate
that.
I was waiting to see if someone could give me an idea how to implement
what
you remembered and, if not, then I would google around to see if I could
find it myself.
Thanks,
Carl
- Original Message - From: André Warnier a...@ice-sa.com
To: Tomcat Users List users@tomcat.apache.org
Sent: Friday, February 12, 2010 3:34 PM
Subject: Re: Tomcat dies suddenly
Carl wrote:
Andre,
Thanks for the response.
I have read almost all of your posts and realy enjoy the way to take
problems apart.
Keep on thinking.
I'm not quite sure how to take the above answer..
So, just in case, and maybe to my own ultimate embarassment, I want to
indicate that I was serious. I seem to remember cases where an
application
at the point of dying, would have very much liked to log a last desperate
message to indicate the situation, but did not even have the resources
left
to be able to do so.
-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat dies suddenly
Chris, I will start the newly rebuilt server with strace tomorrow morning before anyone comes on. Hopefully, strace will yield some useful information. Thanks, Carl - Original Message - From: Christopher Schultz ch...@christopherschultz.net To: Tomcat Users List users@tomcat.apache.org Sent: Friday, February 12, 2010 3:01 PM Subject: Re: Tomcat dies suddenly -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Carl, On 2/12/2010 2:42 PM, Carl wrote: Great ideas (did you see Chris's response with a way of testing the exit code?) [snip] There is no native code in the application (used to do a lot of work in C and I am familiar with mayhem of buffer overruns, pointer screwups, etc.) If you get really desperate, you can also run the jvm inside of strace and get ready for a huge log file. It's possible that you'll see the jvm fail on the same function call every time, and you'll get more information about the problem. strace will show you if a signal terminated the process, or if some other call killed it (like exec(), which would sure do a number on your JVM). - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkt1s58ACgkQ9CaO5/Lv0PB8YQCgq0kuu87WVbPy0P40vFFHyPJW RUsAn1dzTKz2NTm7HAKAK7xeAWJS/2hd =2HBr -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat dies suddenly
Andre, You have the ability to boil things down to the bare essentials. 1) you never saw this issue under a previous JVM 1.5 and Tomcat version 5.5.x Correct. (Running on a P4 with 32 bit Slackware.) 2) the problem happens on two separate servers, which seems to rule out a common server hardware issue Correct. 3) it happens under different versions of Linux, which seems to rule out a problem with one particular Linux distribution Correct... Slackware and openSuse. 4) it seems to be a SegFault in the JVM, leaving a core dump but no traces in the logs. (which SegFaults in my experience happen usually when trying to execute something which is not valid executable code for the platform at hand) Anyway, it does not seem to be due to running out of some resource, nor to a hidden call to system.exit(). Correct... might be some strange code someplace but I can't find any. 5) not quite sure of this anymore, but it seems to happen also on different JVMs, which would tend to rule out a problem with a particular JVM port. No, I have only used Sun's 64 bit. Started with 1.6.0_17 and am now using 1.6.0_18. 6) it does not happen immediately, not in any obvious way related to what is being processsed, except that it seems to happen more readily under load Correct although I am leaning more towards something related to accessing applications B, C and D. Correct that it does not seem to have an issue at any particular point in the code or after some activity by a user. 7) it is obviously not a common problem with either JVM or Tomcat, or we would have had laments from others by now Correct, I think it is something specific to my setup. 8) I don't know how a Java/Tomcat webapp application could trigger a SegFault on its own, other than by having the JVM participate in it. And apparently your apps are working fine up to the moment of the sudden death, so for once they do not appear as being among the usual suspects. Correct. I can see no degradation of speed right up to the moment of failure. 9) This, in one of your earlier posts, triggered my curiosity : quote This Tomcat is straight out of the box except for some modifications to JAVA_OPTS in tomcat/bin/catalina.sh (NDLR: canonically, a better place would be setenv.sh) and opening up ports and turning on SSL in tomcat/conf/server.xml. unquote So, maybe two suggestions, taking into account that I am just making wild guesses here (but that's pretty much what everyone by now is doing too, so I don't feel too bad) : - have you tried running Tomcat from the command-line, with STDOUT/STDERR to the console ? Maybe something shows up there which doesn't show up anywhere else ? I have been starting Tomcat from startup.sh which redirects STDOUT to catalina.out and STDERR to somewhere (I will have to look at it closer.) Starting tomorrow morning, the server which will be running production (I keep the other server in reserve for failures and the old server further back just in case I can't keep up with the failures) will be running under strace to see if that gives us anything (and I will be pounding on applications B, C and D just to see if I can force a failure.) - what about this SSL ? that just seems to me a likely candidate for something that is maybe not used all the time, probably calls stuff which should be native code, and is usually provided separately from Tomcat. Can you turn it off and still be operational ? Also, if it is provided separately, it should probably be relatively grouped in some directory, making it easier to check if everything is as it should be. We use SSL for all communications because most of the data we handle is personal data for children. Can't really turn that off. Note also that apart from a direct hardware similarity between the servers on which it happens, another common element seems to be the place at which it happens, namely the server room. This is a long shot, but a power supply issue may also provoke hardware failures. Or if your server room is on top of a mountain, or near a particle accelerator ? (re relativistic gamma rays, dark energy and all that stuff). ;-) I am not certain but I do know I don't have to use any lights at night, I provide enough glowing (light) to see where I am going. All of servers are on UPS's which are tested periodically. Thanks for your thoughts, you have such a great way of analyzing problems. Carl - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat dies suddenly
Anthony, I have to get through this as quickly as possible and I have never been able to rig up a stress test that duplicates what I am seeing in production so I am basically using the production servers for working the problem out. When a server fails, I just redirect the traffic to the other server and try to analyze what happened. And, if I can't keep the new servers up, I just move back to the old server (thank goodness I didn't rebuild that one when the new ones seemed to work.) Thanks, Carl - Original Message - From: Anthony J. Biacco abia...@formatdynamics.com To: Tomcat Users List users@tomcat.apache.org Sent: Saturday, February 13, 2010 4:08 PM Subject: RE: Tomcat dies suddenly If #1 is correct maybe you should just revert back until you can do more testing outside production. Of course that's only if you're not using some tomcat 6/java 1.6 specific features for your apps -Tony Sent from my Windows® phone. -Original Message- From: André Warnier a...@ice-sa.com Sent: Saturday, February 13, 2010 1:56 PM To: Tomcat Users List users@tomcat.apache.org Subject: Re: Tomcat dies suddenly Carl wrote: Chris, I find it hard to believe two brand new machines with different processors, etc. would have a hardware problem that showed itself in exactly the same way. Further, I have run memTest86 for 30 hours on one of the servers and it showed nothing (although, as Chuck pointed out, the test may not have handled the cores correctly or may not have changed the temperature sufficiently to cause the problem we are seeing.) I have not found a mem test specifically for 64 bit processors. Right. After rescanning your posts (and feel free to correct any discrepancies), here is a summary : 1) you never saw this issue under a previous JVM 1.5 and Tomcat version 5.5.x 2) the problem happens on two separate servers, which seems to rule out a common server hardware issue 3) it happens under different versions of Linux, which seems to rule out a problem with one particular Linux distribution 4) it seems to be a SegFault in the JVM, leaving a core dump but no traces in the logs. (which SegFaults in my experience happen usually when trying to execute something which is not valid executable code for the platform at hand) Anyway, it does not seem to be due to running out of some resource, nor to a hidden call to system.exit(). 5) not quite sure of this anymore, but it seems to happen also on different JVMs, which would tend to rule out a problem with a particular JVM port. 6) it does not happen immediately, not in any obvious way related to what is being processsed, except that it seems to happen more readily under load 7) it is obviously not a common problem with either JVM or Tomcat, or we would have had laments from others by now 8) I don't know how a Java/Tomcat webapp application could trigger a SegFault on its own, other than by having the JVM participate in it. And apparently your apps are working fine up to the moment of the sudden death, so for once they do not appear as being among the usual suspects. 9) This, in one of your earlier posts, triggered my curiosity : quote This Tomcat is straight out of the box except for some modifications to JAVA_OPTS in tomcat/bin/catalina.sh (NDLR: canonically, a better place would be setenv.sh) and opening up ports and turning on SSL in tomcat/conf/server.xml. unquote So, maybe two suggestions, taking into account that I am just making wild guesses here (but that's pretty much what everyone by now is doing too, so I don't feel too bad) : - have you tried running Tomcat from the command-line, with STDOUT/STDERR to the console ? Maybe something shows up there which doesn't show up anywhere else ? - what about this SSL ? that just seems to me a likely candidate for something that is maybe not used all the time, probably calls stuff which should be native code, and is usually provided separately from Tomcat. Can you turn it off and still be operational ? Also, if it is provided separately, it should probably be relatively grouped in some directory, making it easier to check if everything is as it should be. Note also that apart from a direct hardware similarity between the servers on which it happens, another common element seems to be the place at which it happens, namely the server room. This is a long shot, but a power supply issue may also provoke hardware failures. Or if your server room is on top of a mountain, or near a particle accelerator ? (re relativistic gamma rays, dark energy and all that stuff). ;-) - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Re:[OT] Tomcat dies suddenly
Chuck, The cases and even power supplies are very different. The T105 is destined to be a backup server and the T110 is supposed to be the front line guy. Thanks, Carl - Original Message - From: Caldarale, Charles R chuck.caldar...@unisys.com To: Tomcat Users List users@tomcat.apache.org Sent: Saturday, February 13, 2010 4:38 PM Subject: RE: Re:[OT] Tomcat dies suddenly From: André Warnier [mailto:a...@ice-sa.com] Subject: Re:[OT] Tomcat dies suddenly Read relativistic protons instead. Now you're talking about something that can do real damage. (Unlike a WIMP, which seems to be too shy to even show up at the party.) BTW, I was thinking that since the T105 and T110 were both from the same vendor and use the same case, there might be some common design factor causing these mysterious segfaults - but they're radically different on the inside (e.g., AMD vs. Intel, memory speed and type, slots). Not much in common, except perhaps the power supply. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat dies suddenly
Chuck, I am using the mysql-connector/j version 3.1.12. Interesting because the latest driver is version 5.1.11. Is this worth a shot or is it likely to just miuddy the waters? We typically have less than 20 open connections. Thanks, Carl - Original Message - From: Caldarale, Charles R chuck.caldar...@unisys.com To: Tomcat Users List users@tomcat.apache.org Sent: Saturday, February 13, 2010 3:51 PM Subject: RE: Tomcat dies suddenly From: Carl [mailto:c...@etrak-plus.com] Subject: Re: Tomcat dies suddenly We use jdbc with the commons pooling process. Is your JDBC driver type 4, or does it utilize some native code? - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat dies suddenly
Chuck, I started with the default (except for Xms, Xmx and the PermSize settings) and only added the others after the failures started piling up. They are easy to remove and are not likely to be helping or hurting but may be muddying the waters. Thanks, Carl - Original Message - From: Caldarale, Charles R chuck.caldar...@unisys.com To: Tomcat Users List users@tomcat.apache.org Sent: Saturday, February 13, 2010 5:39 PM Subject: RE: Tomcat dies suddenly From: Carl [mailto:c...@etrak-plus.com] Subject: Re: Tomcat dies suddenly This Tomcat is straight out of the box except for some modifications to JAVA_OPTS in tomcat/bin/catalina.sh Have you tried using the default GC mechanism, rather than CMS? (Sorry if I'm repeating something you've already done.) - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat dies suddenly
This problem continues to plague me. A quick recap so you don't have to search your memory or archives. The 10,000 foot view: new Dell T105 and T110, Slackware 13.0 (64 bit), latest Java (64 bit) and latest Tomcat. Machines only run Tomcat and a small, special purpose Java server (which I have also moved to another machine to make certain it wasn't causing any problems.) Periodically, Tomcat just dies leaving no tracks in any log that I have been able to find. The application has run on a Slackware 12.1 (32 bit) for several years without problems (except for application bugs.) I have run memTest86 for 30 hours on the T110 with no problems reported. More details: the Dell 105 has an AMD processor and (currently) 8 GB memory. The T110 has a Xeon 3440 processor and 4 GB memory. The current Java version is 1.6.0_18-b07. The current Tomcat version is 6.0.24. The servers are lightly loaded with less than 100 sessions active at any one time. All of the following trials have produced the same results: 1. Tried openSuse 64 bit. 2. Tried 32 bit Slackware 13. 3. Increased the memory in the T105 from 4GB to 6 GB and finally to 8 GB. 4. Have fiddled with the JAVA_OPTS settings in catalina.sh. The current settings are: JAVA_OPTS=-Xms512m -Xmx512m -XX:PermSize=384m -XX:MaxPermSize=384m -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/usr/local/tomcat/logs I can see the incremental GC effects in both catalina.out and VisualJVM. Note the fairly small (512MB) heap but watching VisualJVM indicates this is sufficient (when a failure occurs, VisualJVM will report the last amount of memory used and this is always well under the max in both heap and permGen.) More information about the failures: 1. They are clean kills as I can restart Tomcat immediately after failure and there is no port conflict. As I understand it, this implies the linux process was killed (I have manually killed the java process with kill -9 and had the same result that I have observed when the system fails) or Tomcat was shut down normally, e.g., using shutdown.sh (this always leaves tracks in catalina.out and I am not seeing any so I do not believe this is the case.) 2. They appear to be load related. On heavy processing days, the system might fail every 15 minutes but it could also run for up to 10 days without failure but with lighter processing. I have found a way to force a more frequent failure. We have four war's deployed (I will call them A, B, C and D.) They are all the same application but we use this process to enable access to different databases. A user accesses the correct application by https://xx.com/A or B, etc. A is used for production while the others have specific purposes. Thus, A is always used while the others are used periodically. If users start coming in on B, C and/or D, within hours the failure occurs (Tomcat shuts down bringing all of the users down, of course.) Note that the failure still does not happen immediately. 3. They do not appear to be caused by memory restrictions as 1) the old server had only 2 GB of memory and ran well, 2) I have tried adding memory to the new servers with no change in behavior and 3) the indications from top and the Slackware system monitor are that the system is not starved for memory. In fact, yesterday, running on the T105 with 8 GB of memory, top never reported over 6 GB being used (0 swap being used) yet it failed at about 4:00PM. 4. Most of the failures will occur after some amount of processing. We update the war's and restart the Tomcats each morning at 1:00AM. Most of the failures will occur toward the end of the day although heavy processing (or using multiple 'applications') may force it to happen earlier (the earliest failure has been around 1:00PM... it was the heaviest processing day ever.) It is almost as if there is a bucket somewhere that gets filled up and, when filled, causes the failure. (So there is no misunderstanding, there has never been an OOM condition reported anywhere that I can find.) Observations (or random musings): The fact that the failures occur after some amount of processing implies that the issue is related to memory usage, and, potentially, caused by a memory leak in the application. However, 1) I have never seen (from VisualJVM) any issue with either heap or permGen and the incremental GC's reported in catalina.out look pretty normal and 2) top, vmstat, system monitor, etc. are not showing any issues with memory. The failures look a lot like the linux OOM killer (which Mark or Chris said way back at the beginning which is now 2-3 months ago.) Does anyone have an idea where I could get information on tracking the linux signals that could cause this? Thanks, Carl - To unsubscribe, e
Re: Tomcat dies suddenly
Chuck, Thanks for your quick reply. I don't think any of the file systems are in danger (we purposely spec'd large disks because they were cheap and we wouldn't have to deal with space shortages.) df gives: Filesystem 1K-blocks Used Available Use% Mounted on /dev/root 76896348468464 72521680 1% / /dev/sda4 66682840 8093552 55201984 13% /usr tmpfs 4085376 0 4085376 0% /dev/shm Tomcat is in /usr/local/tomcat. We have no quotas on any of the file systems (checked with 'quotacheck' just in case some rogue quota was set.) No logs (other than the standard Tomcat logs.) tomcat/temp currently has files totaling 272K... not likely that is a problem. Thanks, Carl - Original Message - From: Caldarale, Charles R chuck.caldar...@unisys.com To: Tomcat Users List users@tomcat.apache.org Sent: Friday, February 12, 2010 8:39 AM Subject: RE: Tomcat dies suddenly From: Carl [mailto:c...@etrak-plus.com] Subject: Re: Tomcat dies suddenly The fact that the failures occur after some amount of processing implies that the issue is related to memory usage, and, potentially, caused by a memory leak in the application. Actually, it's looking less and less like a memory problem here. What about exhaustion of some other resource, such as a disk quota? Do the applications create any kind of log files or otherwise use increasing amounts of disk as they run? - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat dies suddenly
Chris, Thanks for your response. I'll try to find a memtest for 64 bit. On the 32 bit topic. I have run successfully for several years on a Slackware 12.1 32 bit system (Java 1.5 something, Tomcat 5.5.25.) When I brought up Slackware 13.0 32 bit, and the latest Tomcat and Java (32 bit, of course), it suffered from the same problem which surprised me. Also, one of the reasons for going to 64 bit was that we have had problems when some people were running B, C and D (permGen, very clear) so I was trying to get a little extra memory. No, we are not doing remote logging (the servers are 10' away from me.) Me stumped also... has always been so simple to set up a Tomcat server. Do I gain anything by trying Glassfish? Thanks, Carl - Original Message - From: Christopher Schultz ch...@christopherschultz.net To: Tomcat Users List users@tomcat.apache.org Sent: Friday, February 12, 2010 10:13 AM Subject: Re: Tomcat dies suddenly -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Carl, On 2/12/2010 7:20 AM, Carl wrote: The 10,000 foot view: new Dell T105 and T110, Slackware 13.0 (64 bit), latest Java (64 bit) and latest Tomcat. Machines only run Tomcat and a small, special purpose Java server (which I have also moved to another machine to make certain it wasn't causing any problems.) Periodically, Tomcat just dies leaving no tracks in any log that I have been able to find. The application has run on a Slackware 12.1 (32 bit) for several years without problems (except for application bugs.) I have run memTest86 for 30 hours on the T110 with no problems reported. All of the following trials have produced the same results: One last check: have you tried a 32-bit JVM running on either a 32-bit or 64-bit OS? Your memory needs are quire modest, so a 32-bit JVM should suffice. I think you mentioned that, in the past, on lesser hardware, a 32-bit JVM was being used. The failures look a lot like the linux OOM killer (which Mark or Chris said way back at the beginning which is now 2-3 months ago.) While the Linux OOM killer /is/ possible, I agree with Chuck that it's sounding less like a memory problem. I would have bet on a hardware problem, but memtest86 seems to have ruled that out (any x86 system I've suspected of having flaky hardware has always failed on it's first round of memtest86 testing). I wonder if there's a memtest_x64 that you should be trying, instead shrug. I honestly can't think of any other reason why Tomcat would be just dying with no trace: a clean shutdown would produce logs. The Linux OOM killer would say something in the system logs (you're not doing remote syslogging, and the message is somehow getting lost, are you?). A JVM bug would likely generate an hs_pid.log file with details of the crash. I have to admit, I'm stumped. :( - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkt1cAcACgkQ9CaO5/Lv0PB6lgCfc2s6FpRt+LNqzgNV2KG76FiZ VeQAn3utd641tdOet+lQkG+QjpUg/txt =QLU+ -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat dies suddenly
Jeffrey, Thanks for taking the time to give this some thought. 1. There is no place in the code that we intentionally put an exit(). I have grepped for exit() and found nothing. The system stops in a different place every time... the last entry in catalina.out has never been the same (over 15-20 failures.) 2. No, we do not share jars or classes... our thought was this was a potential for screwups and not really gaining anything. 3. Good idea... I will try this. Thanks, Carl - Original Message - From: Jeffrey Janner jeffrey.jan...@polydyne.com To: Tomcat Users List users@tomcat.apache.org Sent: Friday, February 12, 2010 10:31 AM Subject: RE: Tomcat dies suddenly I've been following this thread with interest, but haven't weighed in since I'm doing much development these days. I have to say that I'm agreeing with Chuck and Chris that it is a resource issue - especially since it doesn't appear to be a problem unless under load. Also, the OOP mentioned that it doesn't seem to occur if only the production app is being hit. It takes someone using one of the other copies first to generate the problem. So the questions occur: 1) Are you absolutely positive that there is no code that could be calling exit()? 2) Are you sharing jar files or classes between the apps? If so, place a separate copy in the WEB-INF/lib directory of each webapp, and remove them from the shared location, to insure that you're not having an issue with one app stomping on another. The combined load on a resource in a shared class could be overloading some built-in limit, and that class could be causing the fail without logging an error, or you're not catching the error to log it. 3) You have the disk space, try turning on debug level in all your logging utilities and see what the apps were last doing. Jeff *** NOTICE * This message is intended for the use of the individual or entity to which it is addressed and may contain information that is privileged, confidential, and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient or the employee or agent responsible for delivering this message to the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this communication is strictly prohibited. If you have received this communication in error, please notify us immediately by reply or by telephone (call us collect at 512-343-9100) and immediately delete this message and all its attachments. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat dies suddenly
Tony, Thanks for your response. Way back (several months ago), I did start with an older version of Tomcat. When it showed the failures, I moved forward to the current version with a step in between. I did not try to go back to 5.5 or Java 1.5 (and I think that would be counter productive.) Thanks, Carl - Original Message - From: Anthony J. Biacco abia...@formatdynamics.com To: Tomcat Users List users@tomcat.apache.org Sent: Friday, February 12, 2010 10:36 AM Subject: RE: Tomcat dies suddenly You haven't said if you tried a previous java version or tomcat version (6.0.20)? -Tony Sent from my Windows® phone. -Original Message- From: Carl c...@etrak-plus.com Sent: Friday, February 12, 2010 5:22 AM To: Tomcat Users List users@tomcat.apache.org Subject: Re: Tomcat dies suddenly This problem continues to plague me. A quick recap so you don't have to search your memory or archives. The 10,000 foot view: new Dell T105 and T110, Slackware 13.0 (64 bit), latest Java (64 bit) and latest Tomcat. Machines only run Tomcat and a small, special purpose Java server (which I have also moved to another machine to make certain it wasn't causing any problems.) Periodically, Tomcat just dies leaving no tracks in any log that I have been able to find. The application has run on a Slackware 12.1 (32 bit) for several years without problems (except for application bugs.) I have run memTest86 for 30 hours on the T110 with no problems reported. More details: the Dell 105 has an AMD processor and (currently) 8 GB memory. The T110 has a Xeon 3440 processor and 4 GB memory. The current Java version is 1.6.0_18-b07. The current Tomcat version is 6.0.24. The servers are lightly loaded with less than 100 sessions active at any one time. All of the following trials have produced the same results: 1. Tried openSuse 64 bit. 2. Tried 32 bit Slackware 13. 3. Increased the memory in the T105 from 4GB to 6 GB and finally to 8 GB. 4. Have fiddled with the JAVA_OPTS settings in catalina.sh. The current settings are: JAVA_OPTS=-Xms512m -Xmx512m -XX:PermSize=384m -XX:MaxPermSize=384m -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/usr/local/tomcat/logs I can see the incremental GC effects in both catalina.out and VisualJVM. Note the fairly small (512MB) heap but watching VisualJVM indicates this is sufficient (when a failure occurs, VisualJVM will report the last amount of memory used and this is always well under the max in both heap and permGen.) More information about the failures: 1. They are clean kills as I can restart Tomcat immediately after failure and there is no port conflict. As I understand it, this implies the linux process was killed (I have manually killed the java process with kill -9 and had the same result that I have observed when the system fails) or Tomcat was shut down normally, e.g., using shutdown.sh (this always leaves tracks in catalina.out and I am not seeing any so I do not believe this is the case.) 2. They appear to be load related. On heavy processing days, the system might fail every 15 minutes but it could also run for up to 10 days without failure but with lighter processing. I have found a way to force a more frequent failure. We have four war's deployed (I will call them A, B, C and D.) They are all the same application but we use this process to enable access to different databases. A user accesses the correct application by https://xx.com/A or B, etc. A is used for production while the others have specific purposes. Thus, A is always used while the others are used periodically. If users start coming in on B, C and/or D, within hours the failure occurs (Tomcat shuts down bringing all of the users down, of course.) Note that the failure still does not happen immediately. 3. They do not appear to be caused by memory restrictions as 1) the old server had only 2 GB of memory and ran well, 2) I have tried adding memory to the new servers with no change in behavior and 3) the indications from top and the Slackware system monitor are that the system is not starved for memory. In fact, yesterday, running on the T105 with 8 GB of memory, top never reported over 6 GB being used (0 swap being used) yet it failed at about 4:00PM. 4. Most of the failures will occur after some amount of processing. We update the war's and restart the Tomcats each morning at 1:00AM. Most of the failures will occur toward the end of the day although heavy processing (or using multiple 'applications') may force it to happen earlier (the earliest failure has been around 1:00PM... it was the heaviest processing day ever.) It is almost as if there is a bucket somewhere that gets filled up and, when filled, causes the failure. (So there is no misunderstanding, there has never been an OOM condition reported anywhere that I can find.) Observations (or random musings
Re: Tomcat dies suddenly
Chris,
Thank you... I would never have thought about this script. I'll fire that
baby up tonight.
Thanks,
Carl
- Original Message -
From: Christopher Schultz ch...@christopherschultz.net
To: Tomcat Users List users@tomcat.apache.org
Sent: Friday, February 12, 2010 10:49 AM
Subject: Re: Tomcat dies suddenly
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
Jeffrey,
On 2/12/2010 10:31 AM, Jeffrey Janner wrote:
Also, the OP mentioned that it doesn't seem to occur if only the
production app is being hit. It takes someone using one of the other
copies first to generate the problem. So the questions occur:
1) Are you absolutely positive that there is no code that could be
calling exit()?
One way to check this would be to launch the JVM using a script that
captures the exit code of the process:
$!/bin/sh
# Grab all these variable definitions from bin/catalina.sh
$_RUNJAVA $LOGGING_CONFIG $JAVA_OPTS $CATALINA_OPTS \
-Djava.endorsed.dirs=$JAVA_ENDORSED_DIRS -classpath $CLASSPATH \
-Dcatalina.base=$CATALINA_BASE \
-Dcatalina.home=$CATALINA_HOME \
-Djava.io.tmpdir=$CATALINA_TMPDIR \
org.apache.catalina.startup.Bootstrap $@ start \
$CATALINA_BASE/logs/catalina.out 21
result=$?
echo JVM existed with status code=${result}
If it's 0, then the changes that this is due to a System.exit(0) call
are very high.
You could also try running under a SecurityManager, and prohibit
System.exit() calls.
- -chris
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.10 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iEYEARECAAYFAkt1eKMACgkQ9CaO5/Lv0PBMIACgkZAP4UQQqVpELU2Ej4HQFOLI
7GEAn1rsUxRxvyXBrKGBYlomkaI0WN8C
=hSGU
-END PGP SIGNATURE-
-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat dies suddenly
Peter, Great ideas (did you see Chris's response with a way of testing the exit code?) Of course I am willing to add some debug code (I'll do almost anything at this point)... I will look at the links you provided and try it out. I've contented right along that the major issue is that the failure leaves no tracks... I'm hopeful the 'debug' code you have suggested will allow me to start to understand the underlying cause. I will leave the security manager to last as I don't know that stuff very well. There is no native code in the application (used to do a lot of work in C and I am familiar with mayhem of buffer overruns, pointer screwups, etc.) Thanks, Carl - Original Message - From: Peter Crowther peter.crowt...@melandra.com To: Tomcat Users List users@tomcat.apache.org Sent: Friday, February 12, 2010 12:05 PM Subject: Re: Tomcat dies suddenly On 12 February 2010 16:43, Carl c...@etrak-plus.com wrote: 1. There is no place in the code that we intentionally put an exit(). I have grepped for exit() and found nothing. The system stops in a different place every time... the last entry in catalina.out has never been the same (over 15-20 failures.) I'm wondering about a concurrency issue, given that the failure occurs more frequently under load but can occur at other times. But it's difficult to think of one that would cause a silent crash like this, unless you're using a library somewhere that makes use of the crash and burn school of error handling! ... actually, that's a thought. There *is* a way of you distinguishing the VM exiting in an orderly fashion, quitting, or being terminated, and that's to add a tiny webapp that just registers a shutdown hook (http://java.sun.com/j2se/1.5.0/docs/api/java/lang/Runtime.html#addShutdownHook%28java.lang.Thread%29). If you're willing to add that debug code, then you could log a message (or simply touch a file). If there's no message/file, the VM is shutting down due to an error or someone's calling halt (http://java.sun.com/j2se/1.5.0/docs/api/java/lang/Runtime.html#halt%28int%29). As another small piece of debugging, it would be very interesting to capture the exit status of the JVM. How are you starting it, and is there any chance of inspecting the code on exit? A non-zero exit code, in particular, would be interesting. As a third, rather larger, piece of debugging, you could consider running Tomcat under a security manager that allowed all operations except exit. This may be tough to get right, especially on a production server, but it would definitely tell you whether there were any Java calls to Runtime.exit() or Runtime.halt(). Finally, is there any native code in any part of your application? This is, of course, outside of any of the JVM handlers; a failure of native code can (and occasionally does!) cause mayhem. None of this is a solution, I'm afraid. It's all just more debugging and gathering more information. But the problem is sufficiently unusual that I think you're going to have to keep on debugging it :-(. - Peter - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat dies suddenly
Tony, I tried stressing it with JMeter and came up no results. I could push it hard enough to force an OOM but it performed/failed as expected leaving tracks all over the place. The stressing was not very sophisticated (just a couple of the production jsp's) but, like I said, it didn't show anything (I was really testing to see if the problem was in GC... it wasn't.) Might rig up a more comprehensive test... will see after I try Chris and Peter's ideas. Thanks, Carl - Original Message - From: anthonyvie...@gmail.com To: Tomcat Users List users@tomcat.apache.org Sent: Friday, February 12, 2010 12:07 PM Subject: Re: Tomcat dies suddenly Is it possible to run this server with a basic tomcat application under load to rule out the application causing the crash? On Fri, Feb 12, 2010 at 4:20 AM, Carl c...@etrak-plus.com wrote: This problem continues to plague me. A quick recap so you don't have to search your memory or archives. The 10,000 foot view: new Dell T105 and T110, Slackware 13.0 (64 bit), latest Java (64 bit) and latest Tomcat. Machines only run Tomcat and a small, special purpose Java server (which I have also moved to another machine to make certain it wasn't causing any problems.) Periodically, Tomcat just dies leaving no tracks in any log that I have been able to find. The application has run on a Slackware 12.1 (32 bit) for several years without problems (except for application bugs.) I have run memTest86 for 30 hours on the T110 with no problems reported. More details: the Dell 105 has an AMD processor and (currently) 8 GB memory. The T110 has a Xeon 3440 processor and 4 GB memory. The current Java version is 1.6.0_18-b07. The current Tomcat version is 6.0.24. The servers are lightly loaded with less than 100 sessions active at any one time. All of the following trials have produced the same results: 1. Tried openSuse 64 bit. 2. Tried 32 bit Slackware 13. 3. Increased the memory in the T105 from 4GB to 6 GB and finally to 8 GB. 4. Have fiddled with the JAVA_OPTS settings in catalina.sh. The current settings are: JAVA_OPTS=-Xms512m -Xmx512m -XX:PermSize=384m -XX:MaxPermSize=384m -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/usr/local/tomcat/logs I can see the incremental GC effects in both catalina.out and VisualJVM. Note the fairly small (512MB) heap but watching VisualJVM indicates this is sufficient (when a failure occurs, VisualJVM will report the last amount of memory used and this is always well under the max in both heap and permGen.) More information about the failures: 1. They are clean kills as I can restart Tomcat immediately after failure and there is no port conflict. As I understand it, this implies the linux process was killed (I have manually killed the java process with kill -9 and had the same result that I have observed when the system fails) or Tomcat was shut down normally, e.g., using shutdown.sh (this always leaves tracks in catalina.out and I am not seeing any so I do not believe this is the case.) 2. They appear to be load related. On heavy processing days, the system might fail every 15 minutes but it could also run for up to 10 days without failure but with lighter processing. I have found a way to force a more frequent failure. We have four war's deployed (I will call them A, B, C and D.) They are all the same application but we use this process to enable access to different databases. A user accesses the correct application by https://xx.com/A or B, etc. A is used for production while the others have specific purposes. Thus, A is always used while the others are used periodically. If users start coming in on B, C and/or D, within hours the failure occurs (Tomcat shuts down bringing all of the users down, of course.) Note that the failure still does not happen immediately. 3. They do not appear to be caused by memory restrictions as 1) the old server had only 2 GB of memory and ran well, 2) I have tried adding memory to the new servers with no change in behavior and 3) the indications from top and the Slackware system monitor are that the system is not starved for memory. In fact, yesterday, running on the T105 with 8 GB of memory, top never reported over 6 GB being used (0 swap being used) yet it failed at about 4:00PM. 4. Most of the failures will occur after some amount of processing. We update the war's and restart the Tomcats each morning at 1:00AM. Most of the failures will occur toward the end of the day although heavy processing (or using multiple 'applications') may force it to happen earlier (the earliest failure has been around 1:00PM... it was the heaviest processing day ever.) It is almost as if there is a bucket somewhere that gets filled up and, when filled, causes the failure. (So there is no misunderstanding, there has never been an OOM condition
Re: Tomcat dies suddenly
Konstantin, Your response provides some interesting ideas. Remember that I had tried shutting Tomcat down by killing (kill -9 xxx where xxx is the process id) the java process and saw the same ending in the logs that I have been seeing when the system 'fails'. So it would seem that I could test the implementation of Chris' and Peter's ideas by simply killing the java process and see what the debugging code gives me. At least that looks like a baseline. Thanks for the help. Carl - Original Message - From: Konstantin Kolinko knst.koli...@gmail.com To: Tomcat Users List users@tomcat.apache.org Sent: Friday, February 12, 2010 1:17 PM Subject: Re: Tomcat dies suddenly 2010/2/12 Carl c...@etrak-plus.com: More details: the Dell 105 has an AMD processor and (currently) 8 GB memory. The T110 has a Xeon 3440 processor and 4 GB memory. The current Java version is 1.6.0_18-b07. The current Tomcat version is 6.0.24. Actually, 6.0.24 version *will* exit silently if it is terminated by a signal. To reproduce (on Windows): 1. Run Tomcat in a console window. 2. Press Ctrl+C 3. Tomcat shutdowns in several seconds, cleanly, but silently. That is because in 6.0.24 a VM shutdown hook was added that cleanly terminates logging subsystem (flushing all cached messages). The VM starts all shutdown hook threads at the same time and they run in parallel. Thus if Tomcat is being stopped through a hook, the logging subsystem has a chance to be already shut down at that time. I think that the same silent shutdown will occur if calling System.exit(), by the same reason. I do not know, what will happen if running Tomcat with jsvc on a Unix. (IIRC, jsvc calls the stop method in Bootstrap/Catalina, so Tomcat shuts down not through a hook and all messages should be present). The above is specific to 6.0.24. In 6.0.20 logs are forcibly flushed after each message is written, and log files are never explicitly closed. Best regards, Konstantin Kolinko - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat dies suddenly
Chuck, Oooo, that's interesting. Just tried kill -9 xxx (the process id of the java process) on one the servers not currently in production and it put nothing in the catalina.out file and nothing in the Slackware messages file. Time to test the ideas from Chris and Peter. Thanks for your insights. Carl - Original Message - From: Caldarale, Charles R chuck.caldar...@unisys.com To: Tomcat Users List users@tomcat.apache.org Sent: Friday, February 12, 2010 1:57 PM Subject: RE: Tomcat dies suddenly From: Konstantin Kolinko [mailto:knst.koli...@gmail.com] Subject: Re: Tomcat dies suddenly Actually, 6.0.24 version *will* exit silently if it is terminated by a signal. To reproduce (on Windows): 1. Run Tomcat in a console window. 2. Press Ctrl+C 3. Tomcat shutdowns in several seconds, cleanly, but silently. Not for me; I get the following in catalina.2010-02-12.log after entering CTRL-C: Feb 12, 2010 12:53:42 PM org.apache.coyote.http11.Http11Protocol pause INFO: Pausing Coyote HTTP/1.1 on http-8080 This is Tomcat 6.0.24, JDK 1.6.0_14, Windows XP Pro 32-bit. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat dies suddenly
Andre, Thanks for the response. I have read almost all of your posts and realy enjoy the way to take problems apart. Keep on thinking. Thanks, Carl - Original Message - From: André Warnier a...@ice-sa.com To: Tomcat Users List users@tomcat.apache.org Sent: Friday, February 12, 2010 2:49 PM Subject: Re: Tomcat dies suddenly Carl wrote: .. a lot of messages, and I have not yet read all of them, but I sympathise with the predicament. I am very far from any kind of Java expertise, so just something that I vaguely seem to remember, although it may be from a very long time ago, not applicable, already mentioned by someone, and maybe not relevant at all.. I thus vaguely seem to remember that there existed some option to the JVM to provide a last resort amount of memory, only used if the JVM really ran out of memory, for the purpose of allowing at least some last-ditch desperate action to take place (like logging the catastrophe maybe). I don't remember how it's called or how it works, but maybe it will raise something in some expert's memory. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat dies suddenly
Chris, Another great idea. I can deal with huge log files a whole lot better than continued failures. Thanks, Carl - Original Message - From: Christopher Schultz ch...@christopherschultz.net To: Tomcat Users List users@tomcat.apache.org Sent: Friday, February 12, 2010 3:01 PM Subject: Re: Tomcat dies suddenly -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Carl, On 2/12/2010 2:42 PM, Carl wrote: Great ideas (did you see Chris's response with a way of testing the exit code?) [snip] There is no native code in the application (used to do a lot of work in C and I am familiar with mayhem of buffer overruns, pointer screwups, etc.) If you get really desperate, you can also run the jvm inside of strace and get ready for a huge log file. It's possible that you'll see the jvm fail on the same function call every time, and you'll get more information about the problem. strace will show you if a signal terminated the process, or if some other call killed it (like exec(), which would sure do a number on your JVM). - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkt1s58ACgkQ9CaO5/Lv0PB8YQCgq0kuu87WVbPy0P40vFFHyPJW RUsAn1dzTKz2NTm7HAKAK7xeAWJS/2hd =2HBr -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat dies suddenly
Chuck, Didn't know that but there is nothing in the catalina.2010-02-12.log except the lines from stopping Tomcat at 1:10AM this morning (the scripted restart after rolling out the changes from yesterday.) There is nothing in any of the other logs in the ~/tomcat/logs directory either. Thank you for the help. Carl - Original Message - From: Caldarale, Charles R chuck.caldar...@unisys.com To: Tomcat Users List users@tomcat.apache.org Sent: Friday, February 12, 2010 3:09 PM Subject: RE: Tomcat dies suddenly From: Carl [mailto:c...@etrak-plus.com] Subject: Re: Tomcat dies suddenly Just tried kill -9 xxx (the process id of the java process) on one the servers not currently in production and it put nothing in the catalina.out file and nothing in the Slackware messages file. The message won't be in catalina.out - look in catalina.-MM-dd.log. Also, what Windows does may not be the same as Linux. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat dies suddenly
Chris, That script would also (I think) kill the VisualJVM and my small java server, wouldn't it. That is not happening... those are staying very much alive, just Tomcat goes down. Thanks, Carl - Original Message - From: Christopher Schultz ch...@christopherschultz.net To: Tomcat Users List users@tomcat.apache.org Sent: Friday, February 12, 2010 3:10 PM Subject: Re: Tomcat dies suddenly -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Carl, On 2/12/2010 3:03 PM, Carl wrote: Oooo, that's interesting. Just tried kill -9 xxx (the process id of the java process) on one the servers not currently in production and it put nothing in the catalina.out file and nothing in the Slackware messages file. Time to test the ideas from Chris and Peter. Maybe you should look around for a script like this: #!/bin/sh while true do sleep `random` killall -KILL java done :) - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkt1tb4ACgkQ9CaO5/Lv0PArjgCeJ5HUImbpQCTyBgqVFt16bqUB NtMAn2kuTh159ad00+Y059p3XqCj5tZr =cLAj -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat dies suddenly
Konstantin, When I shut Tomcat down using shutdown.sh, I see these messages (from 1:10AM this morning): Feb 12, 2010 1:10:02 AM org.apache.coyote.http11.Http11Protocol pause INFO: Pausing Coyote HTTP/1.1 on http-8080 Feb 12, 2010 1:10:02 AM org.apache.coyote.http11.Http11Protocol pause INFO: Pausing Coyote HTTP/1.1 on http-8443 Feb 12, 2010 1:10:02 AM org.apache.coyote.http11.Http11Protocol pause INFO: Pausing Coyote HTTP/1.1 on http-443 Feb 12, 2010 1:10:03 AM org.apache.catalina.core.StandardService stop INFO: Stopping service Catalina Feb 12, 2010 1:10:03 AM org.apache.coyote.http11.Http11Protocol destroy INFO: Stopping Coyote HTTP/1.1 on http-8080 Feb 12, 2010 1:10:03 AM org.apache.coyote.http11.Http11Protocol destroy INFO: Stopping Coyote HTTP/1.1 on http-8443 Feb 12, 2010 1:10:03 AM org.apache.coyote.http11.Http11Protocol destroy INFO: Stopping Coyote HTTP/1.1 on http-443 in catalina.-MM-DD.log but I never see any messages when I do kill -9 xxx to kill the java process. Thanks, Carl - Original Message - From: Konstantin Kolinko knst.koli...@gmail.com To: Tomcat Users List users@tomcat.apache.org Sent: Friday, February 12, 2010 3:19 PM Subject: Re: Tomcat dies suddenly 2010/2/12 Carl c...@etrak-plus.com: Chuck, Oooo, that's interesting. Just tried kill -9 xxx (the process id of the java process) on one the servers not currently in production and it put nothing in the catalina.out file and nothing in the Slackware messages file. Time to test the ideas from Chris and Peter. Thanks for your insights. Carl - Original Message - From: Caldarale, Charles R chuck.caldar...@unisys.com To: Tomcat Users List users@tomcat.apache.org Sent: Friday, February 12, 2010 1:57 PM Subject: RE: Tomcat dies suddenly From: Konstantin Kolinko [mailto:knst.koli...@gmail.com] Subject: Re: Tomcat dies suddenly Actually, 6.0.24 version *will* exit silently if it is terminated by a signal. To reproduce (on Windows): 1. Run Tomcat in a console window. 2. Press Ctrl+C 3. Tomcat shutdowns in several seconds, cleanly, but silently. Not for me; I get the following in catalina.2010-02-12.log after entering CTRL-C: Feb 12, 2010 12:53:42 PM org.apache.coyote.http11.Http11Protocol pause INFO: Pausing Coyote HTTP/1.1 on http-8080 This is Tomcat 6.0.24, JDK 1.6.0_14, Windows XP Pro 32-bit. That depends on your luck. As I said, all hooks run in parallel. You can compare those messages with the ones printed when you shutdown through catalina.bat stop (aka shutdown.bat). Also, note that to disable buffering in log subsystem in 6.0.24 you can configure bufferSize=-1 for all your FileHandlers in logging.properties, see https://issues.apache.org/bugzilla/show_bug.cgi?id=48614#c5 6.0.25 and later will have log buffering turned off by default. Best regards, Konstantin Kolinko - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat dies suddenly
Andre, Take my comment as a compliment because that is the way it was meant... you have helped a lot of people on this least and I, for one, really appreciate that. I was waiting to see if someone could give me an idea how to implement what you remembered and, if not, then I would google around to see if I could find it myself. Thanks, Carl - Original Message - From: André Warnier a...@ice-sa.com To: Tomcat Users List users@tomcat.apache.org Sent: Friday, February 12, 2010 3:34 PM Subject: Re: Tomcat dies suddenly Carl wrote: Andre, Thanks for the response. I have read almost all of your posts and realy enjoy the way to take problems apart. Keep on thinking. I'm not quite sure how to take the above answer.. So, just in case, and maybe to my own ultimate embarassment, I want to indicate that I was serious. I seem to remember cases where an application at the point of dying, would have very much liked to log a last desperate message to indicate the situation, but did not even have the resources left to be able to do so. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat dies suddenly
Chris, I was preparing to try out your checking the jvm exit code and noticed (drum roll) that this morning's failure left a core dump in the tomcat/bin directory. Opening it with gdb -c yielded: This GDB was configured as x86_64-slackware-linux. (no debugging symbols found) Core was generated by `/usr/local/java/bin/java -Djava.util.logging.config.file=/usr/local/tomcat/conf'. Program terminated with signal 11, Segmentation fault. This Tomcat is straight out of the box except for some modifications to JAVA_OPTS in tomcat/bin/catalina.sh and opening up ports and turning on SSL in tomcat/conf/server.xml. There is a logging.properties file in tomcat/conf. How does this line: `/usr/local/java/bin/java -Djava.util.logging.config.file=/usr/local/tomcat/conf' know that it is supposed to use the logging.properties file if that is the problem? Or, is there some other problem? Now, this is embarrassing: I just checked the other server and it also has a core file with the date and time of the last failure in the tomcat/bin directory. And, it shows a seg fault at exactly the same code. This might be a winner... we certainly know it killed the java process. Thanks, Carl - Original Message - From: Christopher Schultz ch...@christopherschultz.net To: Tomcat Users List users@tomcat.apache.org Sent: Friday, February 12, 2010 3:01 PM Subject: Re: Tomcat dies suddenly -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Carl, On 2/12/2010 2:42 PM, Carl wrote: Great ideas (did you see Chris's response with a way of testing the exit code?) [snip] There is no native code in the application (used to do a lot of work in C and I am familiar with mayhem of buffer overruns, pointer screwups, etc.) If you get really desperate, you can also run the jvm inside of strace and get ready for a huge log file. It's possible that you'll see the jvm fail on the same function call every time, and you'll get more information about the problem. strace will show you if a signal terminated the process, or if some other call killed it (like exec(), which would sure do a number on your JVM). - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkt1s58ACgkQ9CaO5/Lv0PB8YQCgq0kuu87WVbPy0P40vFFHyPJW RUsAn1dzTKz2NTm7HAKAK7xeAWJS/2hd =2HBr -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat dies suddenly
Chuck, Darn, I thought we were onto something here but, as you suspected, the line contains a lot of parameters and was truncated. So, now, I think we know the JVM is seg faulting, we just don't know where or why. I'm guessing we have to somehow get a stack trace at the point of failure... any idea how I can get one? Or, is there a better way? Thanks, Carl - Original Message - From: Caldarale, Charles R chuck.caldar...@unisys.com To: Tomcat Users List users@tomcat.apache.org Sent: Friday, February 12, 2010 4:52 PM Subject: RE: Tomcat dies suddenly From: Carl [mailto:c...@etrak-plus.com] Subject: Re: Tomcat dies suddenly How does this line: `/usr/local/java/bin/java - Djava.util.logging.config.file=/usr/local/tomcat/conf' know that it is supposed to use the logging.properties file if that is the problem? It doesn't know. The command line parameter should be: -Djava.util.logging.config.file=/usr/local/tomcat/conf/logging.properties Perhaps the display is just a truncation, since there should be several additional command line parameters beyond that one. Use JConsole to see if the full set of properties are present when Tomcat is running normally. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat dies suddenly
Konstantin, Both servers (T105 and T110) have temperature monitoring. I pulled and sent some information (don't remember how I got it) to Dell and they said the system was operating in a normal range. Also, the fact that it occurs on two different, brand new boxes makes me think it is unlikely to be the culprit. Thanks, Carl - Original Message - From: Konstantin Kolinko knst.koli...@gmail.com To: Tomcat Users List users@tomcat.apache.org Sent: Friday, February 12, 2010 5:08 PM Subject: Re: Tomcat dies suddenly Can it be hardware? Do you have ways to monitor temperature in your box? http://www.bitwizard.nl/sig11/ Best regards, Konstantin Kolinko 2010/2/13 Carl c...@etrak-plus.com: Chuck, Darn, I thought we were onto something here but, as you suspected, the line contains a lot of parameters and was truncated. So, now, I think we know the JVM is seg faulting, we just don't know where or why. I'm guessing we have to somehow get a stack trace at the point of failure... any idea how I can get one? Or, is there a better way? Thanks, Carl - Original Message - From: Caldarale, Charles R chuck.caldar...@unisys.com To: Tomcat Users List users@tomcat.apache.org Sent: Friday, February 12, 2010 4:52 PM Subject: RE: Tomcat dies suddenly From: Carl [mailto:c...@etrak-plus.com] Subject: Re: Tomcat dies suddenly How does this line: `/usr/local/java/bin/java - Djava.util.logging.config.file=/usr/local/tomcat/conf' know that it is supposed to use the logging.properties file if that is the problem? It doesn't know. The command line parameter should be: -Djava.util.logging.config.file=/usr/local/tomcat/conf/logging.properties Perhaps the display is just a truncation, since there should be several additional command line parameters beyond that one. Use JConsole to see if the full set of properties are present when Tomcat is running normally. - Chuck - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat dies suddenly
Jonathon, My system is a little different as I don't run Apache and I have another java process running but your script is certainly helpful. Thanks, Carl - Original Message - From: Jonathan Mast jhmast.develo...@gmail.com To: Tomcat Users List users@tomcat.apache.org Sent: Saturday, February 06, 2010 4:38 PM Subject: Re: Tomcat dies suddenly Carl, Here's what I have on my system, you'll obviously need to adjust for your setup, especially the httpd part as I don't believe you're using it: #db-style timestamp STAMP=`date +%F' '%T`; # count the number of httpd child processes AP_COUNT=`ps auxf | grep -c httpd`; # count the number of Tomcat threads. # NOTE: this assumes that the only program that is using java is Tomcat TC_COUNT=`ps auxHS | grep -c bin/java`; # pipe the output of free into grep seaching for the second line MEM_CHECK=`free -m | grep buffers/`; #use a tab delimiter to simplify importing results into a spreadsheet or db TAB=`echo -e '\t'`; MESSAGE=$STAMP$TAB--$TAB$AP_COUNT$TAB$TC_COUNT$TAB$MEM_CHECK$TAB[httpd,tomcat,memory]; echo $MESSAGE; hope you find it helpful On Fri, Feb 5, 2010 at 10:57 PM, chadwickbailey71 chadwickbaile...@yahoo.com wrote: There is no hardware restrictions in 64-bit mode. use 64-bit Linux, this will fix the problem - Learn an http://automatedsocialnetworking.com/ Automated Social Marketing technique WITHOUT Spending More than 5 Minutes Per Month at your Computer :working: -- View this message in context: http://old.nabble.com/Tomcat-dies-suddenly-tp27377457p27476911.html Sent from the Tomcat - User mailing list archive at Nabble.com. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat dies suddenly
Mark, Thanks for your information. I am not certain where the glibc versions I previously gave came from because... as you noted, they are not correct. The correct glibc version is 2.9 and the threading version (I hope I am stating this correctly) is NPTL 2.9. The kernel version is 2.6.29.6. From the Slackware 13.0 release notes, 'We've used the well-tested and recently patched 2.6.29.6 kernel'. I assumed that was about as good a kernel as I could get... was I wrong? I will try LD_ASSUME_KERNEL=2.4.1 (the Slackware site seems to indicate my version supports this setting) on one of the servers (so I can quickly revert to the other one if the setting doesn't work.) I will add the 'XX:ParallelGCThreads=1 option to one of the servers and see how that works. I am moving a litlle slowly because I don't want to make a nbad situation worse and want to be certain I can account for any improvements or screwups. Thanks for your insights. Carl - Original Message - From: Mark Eggers its_toas...@yahoo.com To: Tomcat Users List users@tomcat.apache.org Sent: Saturday, February 06, 2010 9:46 PM Subject: Re: Tomcat dies suddenly --- On Fri, 2/5/10, Carl c...@etrak-plus.com wrote: Carl, 1. The application runs fine on an older system. Do we have the glibc and kernel versions for all systems? The old system: P4. 1GB memory, 1.3GB swap. Uses swap on a regular basis. kernel is 2.4.25. Java is 1.5.0_01-b08. Tomcat is 5.5.23. Glibc is version 2.3..1. New systems: Server A (Dell T110) is a Xeon 3440, sever B (Dell T105) is an AMD. A has 4GB memory and 19GB swap which is never used. B has 6GB memory and 10GB swap which is never used. A and B both use kernel version 2.6.29.6, Java 1.6.0_18-b07 and Tomcat 6.0.24.. Glibc version is 4.3.3 for both A and B. A couple of observations here: Both the old new kernels end in odd numbers. From memory, I thought the odd kernel numbers were experimental, while the even numbers were production or mainline. I don't remember when this numbering system took place, but certainly by the time the 2.6 kernels were released. From kernel.org, I didn't see a 2.6.29 release marked as stable. The thread implementation has changed between the 2.4 and 2.6 kernels. You can see the thread implementation change by running: getconf GNU_LIBPTHREAD_VERSION I'd be interested in knowing the result of that and getconf GNU_LIBC_VERSION on both systems, since I don't recognize 4.3.3 as a glibc version (latest stable is 2.11.1, so I'm assuming 2.4.3.3?). glibc has some thread bugs that were fixed, but not until 2.8 or 2.9. There was also a persistent bug for 32-bit systems that bites Java applications (not your concern since you're running 64-bit) that wasn't fixed until 2.10.1. So in short, I'm guessing this may be a glibc NPTL issue. There are some observations that don't match, in that you're using Java 6 (most problems are reported with Java 1.4 and Java 5), and that you've used OpenSuSE (kernel, glibc version?) with the same Tomcat failure. However: For some of the earlier 2.6 kernels, you could get around NPTL problems by setting this environment variable: export LD_ASSUME_KERNEL=2.4.1 which forces the use of the old linuxthreads model. I don't know if that option is available with the 2.6 kernel that you are using. Another work-around has been posted on the Java bugs forum, albeit for a different threading problem and Java 5: -XX:ParallelGCThreads=1 sets GC to single threads. It's not fixed in the Java bugs database, because later versions of RedHat Linux don't exhibit the SIGSEGV problem. Some people report that single-threading GC solves their problems, while other people report that it doesn't. Some things to try I guess: 1. export LD_ASSUME_KERNEL=2.4.1 (maybe in startup.sh?) if your kernel supports this.. 2. set -XX:ParallelGCThreads=1 in catalina.sh (JAVA_OPTS). This is an experimental switch, not documented here: http://java.sun.com/javase/technologies/hotspot/vmoptions.jsp, but documented here: http://java.sun.com/javase/technologies/hotspot/gc/gc_tuning_6.html 3. Move to an even-numbered kernel with a glibc of 2.10.1 or better. 2.10 might be OK for your environment since the bug fixed in 2.10.1 causes problems for 32-bit systems only. just my two cents . . . . /mde/ - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat dies suddenly
Mark, Since I don't understand what is causing the problem, all information is helpful and I appreciate you taking time to think about what could be wrong. 1. The application runs fine on an older system. Do we have the glibc and kernel versions for all systems? The old system: P4. 1GB memory, 1.3GB swap. Uses swap on a regular basis. kernel is 2.4.25. Java is 1.5.0_01-b08. Tomcat is 5.5.23. Glibc is version 2.3.1. New systems: Server A (Dell T110) is a Xeon 3440, sever B (Dell T105) is an AMD. A has 4GB memory and 19GB swap which is never used. B has 6GB memory and 10GB swap which is never used. A and B both use kernel version 2.6.29.6, Java 1.6.0_18-b07 and Tomcat 6.0.24. Glibc version is 4.3.3 for both A and B. 2. Different usage patterns (?) seem to cause the outages at different rates (if I remember an account of one Friday). What paths in the application were being exercised most heavily during that time? The outages appear to be most frequent in times of heavy transaction processing. This part of the application is basically a shopping cart although the path from start to 'in the cart' has many variations depending on the type of item being registered for, i.e., the registration process steps through 20+ classes each of which could request additional processing, display a screen for user input, etc. It seems during periods of heavy transactions, the system fails more frequently but it may be that the application requires a certain cumulative amount of activity before it fails and that activity can be spread over several hours or several days. However, since Tomcat is restarted once a day (around 1:00AM after rolling out changes), it would seem that the application would not be able to carry activity from one day into the next. Therefore, it would seem that the failure is triggered by something on the day it occurs. Thanks for your help. Carl - Original Message - From: Mark Eggers its_toas...@yahoo.com To: Tomcat Users List users@tomcat.apache.org Sent: Thursday, February 04, 2010 7:10 PM Subject: RE: Tomcat dies suddenly --- On Thu, 2/4/10, Caldarale, Charles R chuck.caldar...@unisys.com wrote: 6. Carl was using 32-bit Linux, which he isn't :( Correct, which made the whole point moot, so I'm not sure why Dan even brought it up. I just mentioned the 32-bit Linux behavior for completeness. I did state that I realized 32-bit Linux is not in play. AFAIK, 64-bit Linux has a wide-open memory addressing scheme. Maybe it considers everything under 17 billion GiB to be low memory, now :) No, the hardware restrictions don't exist in 64-bit mode. This is what I've read as well. If you use 64-bit Linux, this problem goes away. There are also some ways to build the 32-bit kernel in order to reduce this problem. All this is moot since a 64-bit Linux kernel is being used. As to the copy-on-write behavior for fork()d processes, it would help if I read the man pages: Under Linux, fork() is implemented using copy-on-write pages, so the only penalty that it incurs is the time and memory required to duplicate the parent’s page tables, and to create a unique task structure for the child. It turns out that things are a little bit more complicated than that, in that since version 2.3.3 fork is actually a wrapper to clone(2) with the appropriate flags to give the same result as a traditional fork(2) call. All of this is moot however if there is no Runtime.exec() call in the application. I'm a bit curious though about several points: 1. The application runs fine on an older system. Do we have the glibc and kernel versions for all systems? 2. Different usage patterns (?) seem to cause the outages at different rates (if I remember an account of one Friday). What paths in the application were being exercised most heavily during that time? As for cache / buffer / free - I've seen cases where the cache did not go to 0, but swap was in play (slow disk, small amount of memory). Sorry for chasing down the rabbit hole . . . /mde/ - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat dies suddenly
Bill, ulimit -a shows the following: core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited file size (blocks, -f) unlimited pending signals (-i) 40960 max locked memory (kbytes, -l) 64 max memory size (kbytes, -m) unlimited open files (-n) 1024 pipe size(512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 stack size (kbytes, -s) 8192 cpu time (seconds, -t) unlimited max user processes (-u) 40960 virtual memory (kbytes, -v) unlimited file locks (-x) unlimited This looks fine except the 'open files'. I don't think we would exceed that number but it might be possible. Thanks for your interest. Carl - Original Message - From: Bill Stoddard wgstodd...@gmail.com To: Tomcat Users List users@tomcat.apache.org Sent: Friday, February 05, 2010 9:09 AM Subject: Re: Tomcat dies suddenly On 2/5/10 6:31 AM, Carl wrote: Mark, Since I don't understand what is causing the problem, all information is helpful and I appreciate you taking time to think about what could be wrong. Could one of the log files written by Tomcat be bumping into your file size ulimit? ulimit -a to check file size ulimit If the filesize ulimit is not unlimited, compare the value with the size of the files being written by Tomcat. Bill - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat dies suddenly
Peter, Do you see any harm in just doubling the number (to 2048) just to see if it has an impact? Thanks, Carl - Original Message - From: Peter Crowther peter.crowt...@melandra.com To: Tomcat Users List users@tomcat.apache.org Sent: Friday, February 05, 2010 9:58 AM Subject: Re: Tomcat dies suddenly On 5 February 2010 14:41, Carl c...@etrak-plus.com wrote: Bill, ulimit -a shows the following: [...] open files (-n) 1024 [...] This looks fine except the 'open files'. I don't think we would exceed that number but it might be possible. Note that a socket uses a file descriptor in UNIX, so you should include open sockets in that calculation. - Peter - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
