When tomcat freezes silently, what is the best way to look inside the frozen
tomcat or java instance and see the cause? We have 1500+ tomcat instances, and
most of them run great most of the time. But a few times per week, an instance
will become unresponsive. There is usually nothing
Dan said:
Take a full thread dump of the PID. Linux kill -3, if
Windows CTRL+BREAK if you have a console window open.
Mark said:
2. Thread dumps
So I tried kill -3 but it does nothing. The process ignores me like it owes me
child support. I notice that it is in sleep state S1 if that
RTFM
kill -3 doesn't kill the process it generates a thread dump to stdout.
Mark
Ha! It's probably been 30 years since the last time anyone told me to RTFM.
Fair enough, I suppose I deserved it, although in my defense I did check the
man page first. :-)
Thanks!
--Eric
Disclaimer
... and a link to TFM, if Eric didn't see [the other] Mark's response:
http://wiki.apache.org/tomcat/HowTo#How_do_I_obtain_a_thread_d
ump_of_my_running_webapp_.3F
- -chris
I did see that and I even read it before posting. It said kill -3 sends a
SIGQUIT. I guess I jumped to the
You have to balance that against the minimal cost of today's
memory (even ECC RAM is under $10 per GiB).
True, RAM is relatively cheap, but servers are not. We like to stack as
many instances of tomcat on a server as possible while maintaining good
performance. Some of our 8-core 32GB
What are the possible downsides of setting a low initial
memory pool
and a high max pool? If a tomcat app usually needs
approximately 64MB
of heap space, but sometimes as much as 300-400MB, would it
cause any
problems to set the initial pool to 16M and the max pool to 512M?
An
Robinson, Eric wrote:
We have many servers that have been running 100-200 instances of
tomcat each for years without any performance problems.
Most of our servers are Linux 8-core machines with 32GB
RAM, with the
tomcat instances configured with -Xms16M -Xmx192M.
We also have some
If your application
needs 64MB of
Heap space and you allocate only -Xms16M, then right at the
start the
JVM will have to increase the Heap to 64MB (minimum); so
why would you
do that ?
64MB was just a number I threw out. The app actually uses about 20MB at
startup, so we might
Note that you are talking of memory pool, which is a bit
vague. The -Xms and -Xmx parameters relate to how big the
Heap is, which is only one part of the memory space needed by the JVM.
I am just using the terms that I see on the screen when I pull up
tomcat6w.exe.
--Eric
Disclaimer
I can see the lure of only taking what you need and allowing the
JVM to automatically re-size the memory space:
that way, you only take up a huge chunk of memory during peak load
and not all the time.
But why?
If you are going to need, say, 512MiB at peak load, you're
going to
We have many servers that have been running 100-200 instances
of tomcat each for years without any performance problems.
Most of our servers are Linux 8-core machines with 32GB RAM,
with the tomcat instances configured with -Xms16M -Xmx192M.
We also have some Windows servers with 100-150
What are the possible downsides of setting a low initial memory pool and
a high max pool? If a tomcat app usually needs approximately 64MB of
heap space, but sometimes as much as 300-400MB, would it cause any
problems to set the initial pool to 16M and the max pool to 512M?
--
Eric
We have a Windows 2003 R2 x64 virtual machine running as a guest under
Oracle VirtualBox on 64-bit Linux. The Windows VM has 16GB of RAM and is
running approximately 80 separate instances of tomcat. (These instances
do a specific small function and do not require much power or cause much
load.)
Tomcat 6 on our Windows 2003 R2 x64 server runs fine for a day or two,
then silently dies without leaving any messages in the log files. Then
when we try to restart it, we get a Windows error 1067 and the service
will not start. We have to reboot the whole server and then tomcat will
work fine
Tomcat 6 on our Windows 2003 R2 x64 server runs fine for a
day or two,
then silently dies without leaving any messages in the log
files. Then
when we try to restart it, we get a Windows error 1067 and
the service
will not start. We have to reboot the whole server and then tomcat
So are nfs mounts being used? If so, my guess is that the
mount isn't available when Tomcat starts which triggers the undeploy.
Mark
Sadly, this turns out not to be the case. We modified the tomcat init
script so that it waits for the NFS mount to be available before trying
to start the
suppose that what you want to mount there, is the exported NFS
filesystem server01:/some/exported/fs. If you place a
small file
not.yet.mounted inside of the /mnt/nfs01 local
directory, it will
be visible, as long as the mount has not yet really happened.
I'm under the
This is really getting frustrating. We have 75 instances on tomcat6
running on the same server. Each instance runs from its own discrete
directory, where the tomcat-env.sh script and catalina.properties file
set environment vars to point to that unique instance's folders. Every
time we reboot the
On 22/03/2011 17:04, Robinson, Eric wrote:
Context docBase=/ha_ftp.nfs/site001/mobiledoc
debug=1 reloadable=false
/Context
So are nfs mounts being used? If so, my guess is that the
mount isn't available when Tomcat starts which triggers the undeploy.
Mark
--
On 22/03/2011 17:04, Robinson, Eric wrote:
Context docBase=/ha_ftp.nfs/site001/mobiledoc
debug=1 reloadable=false
/Context
So are nfs mounts being used? If so, my guess is that the
mount isn't
available when Tomcat starts which triggers the undeploy
--
Eric Robinson
#!/bin/bash
#
# httpdStartup script for the Apache HTTP Server
#
# chkconfig: - 85 15
# description: Apache is a World Wide Web server. It is used
to serve \
# HTML files and CGI.
# processname: httpd
# config: /etc/httpd/conf/httpd.conf
#
Eric Robinson
#!/bin/bash
#
# httpdStartup script for the Apache HTTP Server
#
# chkconfig: - 85 15
# description: Apache is a World Wide Web server. It is
used to serve
\
# HTML files and CGI.
# processname: httpd
# config: /etc/httpd/conf/httpd.conf
Not necessarily. I did an rpm -qlp on
tomcat-6.0.18-0.noarch.rpm. All
the files it installs are in /opt/tomcat
The files, yes. But it is highly likely that it installs
also links from /etc/tomcat, /usr/share/tomcat,
/usr/lib/tomcat, /var/lib/tomcat and so on.
At least, that is
From looking at the server, it would appear that tomcat was
installed
using tomcat-6.0.18-0.noarch.rpm.
Now you're in trouble. The 3rd-party repackaged versions of
Tomcat typically scatter files all over, and then try to get
things back together with symlinks. You're better off
Hey, here's a quick question (and possibly a stupid one, but I have a thick
skin). If I have a working tomcat6 server, can I install tomcat6 on a new
server by simply copying a few files and directories over from the working
server to the new one and setting permissions?
--Eric
From: Robinson, Eric [mailto:eric.robin...@psmnv.com]
Subject: Installing Tomcat the Brute Force Way?
If I have a working tomcat6 server, can I install tomcat6
on a new
server by simply copying a few files and directories over from the
working server to the new one and setting
I should be able to copy /opt/tomcat and /usr/java to the
new server,
create the tomcat user, set permissions, and be on my way. No?
Don't know about /usr/java, since that usually involves
symlinks; you should probably do an actual JRE or JDK install
for that. /opt/tomcat should be
They are deleted as part of the autoDeploy process when Tomcat
detects that the associated directory or WAR file is deleted. Turn
off autoDeploy or stop deleting your WARs/directories.
Why would rebooting cause this? I'm not aware of anything
that we are
deleting.
You tell us.
On 10/03/2011 14:33, Robinson, Eric wrote:
The only filesystem involved is the local ext3 one that
tomcat lives on.
I can stop and start tomcat as many times as I want and
everything is
fine. But when I reboot the server, the next time tomcat
comes up the
XML files get deleted
The only filesystem involved is the local ext3 one that
tomcat lives on.
I can stop and start tomcat as many times as I want and
everything is
fine. But when I reboot the server, the next time tomcat
comes up the
XML files get deleted. This only happens to the 70 or so
instrances
We have several instances of tomcat running on a server and everything
seems to run fine. We can stop and start all instances of tomcat
independently.
HOWEVER, when we reboot there server, we find that several XML files
have been deleted from the conf directories for each instance. These
files
-Original Message-
From: Mark Thomas [mailto:ma...@apache.org]
Sent: Wednesday, March 09, 2011 3:42 PM
To: Tomcat Users List
Subject: Re: why does tomcat6 delete xml files from
../tomcat6/conf/Catalina/localhost directory?
On 09/03/2011 23:15, Robinson, Eric wrote:
We have
-Original Message-
From: Stijn Vanhoorelbeke [mailto:stijn.vanhoorelb...@gmail.com]
Hi,
I'm using Solr on my Tomcat 6 system. How can I measure how
much sessions there are active?
I could find such info through Tomcat manager
(manager/html/sessions?path=/solr). But that
Disregard that last email. If I had paid attention, I would not have
skipped right past the part where it was about Solr. :-|
--
Eric Robinson
Disclaimer - February 11, 2011
This email and any files transmitted with it are confidential and intended
solely for Tomcat Users List. If
Saw this statement on an IBM web site:
Most JVMs grow towards the upper heap limit (-Xmx/-mx options) when
more memory is required, and do not return memory to the operating
system, even if the memory is no longer needed, until the JVM process
terminates.
Is this true? If thought returning
Thanks for the input, everyone. I see there is mostly agreement on this
question. Once the OS allocates memory to the JVM, the OS never gets it
back. GC does not return memory to the OS.
This is unfortunate behavior for those of us who stack multiple
instances on tomcat/java on the same server as
I would like to set all JVMs to -Xms=32M -Xmx=512M and let
the system
figure out how much memory each instance really needs.
That should work, at the expense of some thrashing of the
heap size as the load waxes and wanes.
As I understand it, it would actually not work because
On 2/8/2011 12:27 PM, Robinson, Eric wrote:
I would like to set all JVMs to -Xms=32M -Xmx=512M and let
the system
figure out how much memory each instance really needs.
That should work, at the expense of some thrashing of the
heap size
as the load waxes and wanes.
As I
As I understand it, it would actually not work because
instances that
peak at 512M but then go back down to, say, 128M, would still never
return the unused memory to the OS so it can be used by
other instances.
Not true - that's the whole point of the -Xms setting. If GC
can
pid's using a mail client that uses header information to
track threads.
So, if you started this thread by hitting reply to an
existing topic then changed the subject and content, the
header thread tracking info will still be present. Thus, to
him, it will appear as though you
Chris,
On 2/8/2011 3:16 PM, Robinson, Eric wrote:
This is great feedback, Chuck. Right now we have 200+ instances
running on the same server, most of which are set to 64MB
min and max
heap. Like I daid, we do not see OOM messages or paging,
and top shows
about 27GB of 32GB in use
Andre,
Like, what if they come back next week and say that now they
need 1 GB heap per instance ?
Sometimes I lose sleep thinking about that scenario. :-)
--Eric
Disclaimer - February 8, 2011
This email and any files transmitted with it are confidential and intended
solely for
So we definitely know the process is gone and not that
it is only no longer responding?
I'm not 100% certain. I did...
netstat -an|grep :3057|grep LISTEN
..and nothing came back. From that we at least know that the process
was no longer listening on its assigned port.
Then I
Intuitively, I would have never thought that it was
/possible/ to run 160 or so instances of Tomcat on
anything but some type of very expensive hardware.
I assure you that it works admirably! And each of the instances runs a
sophisticated medical application with hundreds of JSPs and class
364600 491072 java5101 --Here's the occasional bad boy
604780 746120 java 31131 --Here's the occasional bad boy.
Those sizes appear to be in Kbyte, so we are talking in one
case of a process with a resident size of 360 MB, in the other of
600 MB.I guess thus, that
Yes. Each instance serves a different set of users
in a local office somewhere.
By that I mean each pair of load-balanced instances serves a set of
users in a local office somewhere.
--
Disclaimer - July 27, 2010
This email and any files transmitted with it are confidential and intended
this particular set of users of this complex application, uses
a particular functionality of the application, used by no other
set of users, and that particular functionality contains
(or triggers) a bug that blows away the server.
That seems like the reasonable conclusion. The next
2) Use System.getRuntime().addShutdownHook() to trigger your
own thread when the JVM does decide to exit.
Assuming the OP does not have easy access to the running
application, would that mean adding another application
which is launched at Tomcat start, and runs the above ?
Or do you
What I was trying to say above, in a devious but humoristic
way, is that there are not a lot of people on this list
running those versions anymore.
Fair enough. I realize it's old, but it still works great for 163 of the
164 instances on this server. However, an upgrade is coming soon and
What kind of machine are you running these 163/164
instances of Tomcat on, and when you are running them,
what /does/ free say ?
I have two different servers with 164 instances of tomcat. Both servers
have 2x quad-core 2.8Ghz Xeon processors with 32GB RAM. On the first
server (app03), most
We've observed tomcat5 shutting down seemingly by itself recently. This
morning I saw this in the log:
- Scheduler DefaultQuartzScheduler_$_NON_CLUSTERED shutting down.
Any way to tell why tomcat is shutting down?
--
Eric Robinson
Disclaimer - July 23, 2010
This email and any files
Tomcat 5.x.y what? Be specific. Also tell us the JVM
you're using and the platform you're running on.
Apache Tomcat/5.0.28
Java 1.4.2_09-b05 Sun Microsystems Inc.
Linux 2.6.18-92.1.22.el5PAE i386
If one of your webapps is doing something antisocial such
as calling System.exit(), that
Maybe something from this recent discussion applies:
http://marc.info/?t=12767355721r=1w=2
Good thread, thanks. Unfortunately, the situations it describes don't
apply in this case.
--
Eric Robinson
Disclaimer - July 23, 2010
This email and any files transmitted with it are confidential
This Tomcat and Java are just trying to say, in the only
way they can, that after some 10 years of good and hard
labor, it is now time to retire them and have the burden
taken over by a new generation.
Yes, that's probably it. One of the...
[r...@app03 ~]# ps ax|grep java|wc -l
164
whoever has a user login on the server can have their own Tomcat
playground even though there is just one Tomcat installed?
Pretty much, yes.
--Eric
Disclaimer - June 24, 2010
This email and any files transmitted with it are confidential and intended
solely for Tomcat Users List. If you
You can monitor the gc with jstat.
jstat -gc pid 10s
This wil show you the memory usage of a java instance with the time
spent in GC.
If it does 0.9 sec. of GC every sec. yare running inefficient. :-)
Thanks. I guess I could restate the question as, Does GC inefficiency
really matter if
This is a similar question to one already being discussed in the list
with the subject Setting the Right Amount of Memory.
We have 160 instances of tomcat on the same server, with most instances
configured to use 64-96MB of RAM. We carefully watch the logs for OOMEs.
If we see any, we increase
In top, my java processes all show an average VIRT size of about 250MB
and an average RES size of about 150MB. Most of them were started with a
64MB heap size. I have two questions:
1. Top shows 0k of swap usage, so the system is not swapping. In that
case, why is there a difference between the
2. Where does the 64MB of java heap show up?
Buried inside the VIRT number.
For example, I have a tomcat configured to use 96MB of heap (export
JAVA_OPTS=-ms96M -mx96M). Top shows VIRT=336396, RES=227264. I'm
guessing that the 96MB of heap is buried in BOTH the VIRT and RES
numbers?
--
Eric
Also, I believe VIRT includes memory shared with other
processes, so if you have 50MiB of Java system classes
loaded and a modern JVM which shares them among running JVMs,
then you'll see that 50MiB included in every process's VIRT
that is sharing it, which is somewhat misleading.
Has anyone ever placed an application and its content on a redundant
DFS solution?
So as when one DFS server fails, another takes over.
Does anyone see possible problems with this setup?
ie. when dfs server fails does tomcat loose connection to the app or
is the failover fast enough.
So what is the impact say (picking some random
numbers of the air) if you have an app that is quite
happy with 256MB of heap space but has 1024MB allocated?
My question exactly!
--
Eric Robinson
Disclaimer - June 21, 2010
This email and any files transmitted with it are confidential
Sorry for not responding yet. Busy at what I actually
get paid for, and trying to generate some real numbers
for the question at hand. More later.
No worries, I just want to keep the thread alive until something
definitive comes from it if possible. :-)
--
Eric Robinson
Disclaimer -
Having a borderline heap size can, in the worst case, result
in almost continual GC activity, if there is only room to
allocate a minimal number of objects between GC occurrences.
For what it's worth, either this is not the case in our real-world
situation or the effect is negligible. Even
For what it's worth, either this is not the case in our
real-world situation or the effect is negligible.
Not surprising - you'd have to be very unlucky to be right
at the edge and see a lot of GC activity and be able to
continue running. Usually you'll be over the edge a bit,
On 17/06/2010 08:59, Robinson, Eric wrote:
If your heap size is right on the edge of your minimum for a Tomcat
instance, you may be doing more GC work than is really needed.
However, if you're satisfied with the response time and CPU
utilization, you should be ok.
Time to hit the vendor
Just a note here : 160 X 512 MB = 81 GB. If each Tomcat's
JVM Is allowed to use up to 512 MB of Heap, there might be
moments where a lot of JVM's will be using close to that
amount. Unless your system can really support that amount
of real RAM, you may be in for massive swapping.
If your heap size is right on the edge of your minimum for a Tomcat
instance, you may be doing more GC work than is really needed.
However, if you're satisfied with the response time and
CPU utilization, you should be ok.
My thoughts exactly. Just wanted to check it with the community.
Just wondering, what tools do you use to manage all the
instances? Also, what do you use to look for the OutOfMemory
in logs? I am looking at Splunk too.
Just shell scripts. We have scripts that...
-- set up a new instance and customize it for a new customer
-- stops,
Time to hit the vendor around the head with the cluebat.
If the app is happy with less heap space then increasing
it is only going to cause problems - mainly that GC when it
happens will take longer and trigger longer pauses. You can
mitigate this with GC config (later VMs may make the
We run 150+ instances of tomcat on one server. To optimize memory
utilization, each tomcat instance is configured with 64MB by default
(export JAVA_OPTS=-ms64M -mx64M). We then watch for
Java.Lang.OutOfmemory errors in the logs. If we see any of these over
time, we allocate additional RAM in
I want to run multiple instances of tomcat on the same server, one
instance for each time zone. I have all the separate tomcats running
fine, but they are all using the server's time zone. I've been Googling
and have not found an effective way to do it. Can someone please tell me
how?
--Eric
Never mind, got it. I was using JAVA_OPTS instead of CATALINA_OPTS.
--Eric
Disclaimer - November 24, 2009
This email and any files transmitted with it are confidential and intended
solely for Tomcat Users List. If you are not the named addressee you should not
disseminate, distribute,
I would add that one good place to /start/ looking,
is the DNS name resolution of your customer's workstations.
Because if that is not working properly, then your server
won't even see the request for a while after they click..
Thanks for the suggestion. The clients connect by IP address
That's very helpful, Chris.
The bytes are not guaranteed to have arrived at the client by the time
the valve computes the elapsed time.
Right, but is it generally correct to say that the response time value
represents the time from when the request was received to the time the
response was
Chris,
Your question makes me think that perhaps what you're
seeing is a timing-out keep-alive request:
It's not what I'm seeing in the logs, it's what the customer is seeing
in the application. One of my customers says they are frequently seeing
long application delays on all of the
We added the %D parameter to the AccessLogValve tag in our server.xml
file and now all entries in the jasper logs contain a field showing the
response time in milliseconds. That's very cool.
I just want to be sure of one thing. Does the value represent the total
time from the moment tomcat saw
We have 60 instances of tomcat 5.X running on an RHEL4 server. Each instance
runs a copy of basically the same application, each for a different customer.
According to their respective status pages, the tomcats are all configured to
use a maximum of 64MB memory, but most peak at around 48MB.
Welcome to paging.
I understand the difference between virtual image size and resident
size. The part I don't understand is the relationship between the tomcat
memory configuration as reported by the tomcat status page and the
actual usage as reported by top. Tomcat says it is configured to
Welcome to paging.
I understand the difference between resident size and virtual image.
What I don't understand is why the tomcat status page indicates that
tomcat is configured to use a maximum of 64MB, but the resident size is
twice that and the virual image is four times that. In other
That's 64 MB for the *Java heap*, not the entire process.
Thanks, that's what I didn't know.
--
Eric
Disclaimer - December 15, 2008
This email and any files transmitted with it are confidential and intended
solely for Tomcat Users List. If you are not the named addressee you should not
How would be go about implementing the following kind of access scheme?
1. If the user's IP address is in an approved range, let them connect to
tomcat anonymously.
2. If the IP address is not in an approved range, authenticate them
through Active Directory.
Is that even possible?
--
Eric
I'm too lazy for all that. I was hoping it would amount a few extra
lines in some configuration file. Now I'm thinking I'll figure out a way
to lock it down in group policy or with some changes to the proxy
server.
--
Eric Robinson
Disclaimer - May 13, 2008
This email and any files
We have a medical program that runs on two tomcat servers behind an LVS
load balancer. Everything works fine except for one particular function
that apparently relies on data in the tomcat5/work directory. If the
load balancer directs a request to a tomcat server that has different
cache data, the
: How to Disable Caching?
Robinson, Eric wrote:
We have a medical program that runs on two tomcat servers behind an
LVS load balancer. Everything works fine except for one particular
function that apparently relies on data in the tomcat5/work directory.
If the load balancer directs a request
or to
Fidelity may be subject to our monitoring procedures. 'Direct link to
Fidelitys website.
http://www.fidelity-international.com/world/index.html
-Original Message-
From: Robinson, Eric [mailto:[EMAIL PROTECTED]
Sent: 01 August 2006 14:03
To: Tomcat Users List; Corobitsyn Roman
Subject: RE
Please accept my apologies for being so direct, but we're in a huge
hurry to get the answer to a tomcat configuration problem. I think it is
a simple problem, but I have been unable to figure it out. (See my post
earlier today entitled, Someone Please: Why Is Tomcat Looking In The
Wrong
AM
To: Tomcat Users List
Subject: Re: Someone Please: Why Is Tomcat Looking In The Wrong
Directory?
On 8/1/06, Robinson, Eric [EMAIL PROTECTED] wrote:
mobiledoc.xml exists in /usr/local/tomcat5/conf/Catalina/localhost
mobiledoc.xml exists in
/home/myaccount/tomcat5/conf/Catalina/localhost
Robinson, Eric wrote:
Hello all,
When I start tomcat, I get several messages like these:
java.io.FileNotFoundException:
/usr/local/tomcat5/webapps/myapp/conf/myapp.properties (No such file or
directory)
This file actually exists in
/home/myaccount/tomcat5/webapps/myapp/conf/myapp.properties
[mailto:[EMAIL PROTECTED]
Sent: Tuesday, August 01, 2006 8:08 AM
To: Tomcat Users List
Cc: Robinson, Eric
Subject: Re: Someone Please: Why Is Tomcat Looking In The Wrong
Directory?
On Aug 1, 2006, at 10:15 AM, Robinson, Eric wrote:
The contents of the file are as follows:
Context path=/mobiledoc
or a relative path to the current directory at the time tomcat was
started.
--David
Robinson, Eric wrote:
Sorry, I cannot answer that question as I am not the developer.
Although I called it myapp for simplicty, it is really a third-party
application.
--
Eric Robinson
-Original Message
Chris,
It makes sense, but before I try the symlink approach, be advised of the
full scenario. Multiple doctor's offices will be connected to the same
server. Each office will have its own instance of tomcat running a copy
(sometimes a different version) of the 3rd party app. The config files
are
Please: Why Is Tomcat Looking In The Wrong
Directory?
Hmmm. and the plot thickens. Have you asked the vendor about this
issue? Can they offer any insights as to how they access the
myapp.properties file?
--David
Robinson, Eric wrote:
David,
I do not know if the 3rd party app has hard
93 matches
Mail list logo
