Sam Hokin wrote:
André Warnier wrote:
What in this directory path is (are) a symlink(s) ?
/home/sites/ims/jcms/webapps/ROOT/WEB-INF/classes/org/apache/jsp/test6_jsp.class
There are no symlinks in that path.
Then the question is : why is that thing doing lstat's ?
(I mean in the trace list
Caldarale, Charles R wrote:
I'll poke around in the webapp classloader to see if I can find anything
interesting, but in the meantime, is there a /net directory on the problematic
server? If there is and it targets a remote file system, that might explain
the long delay on these stat()
Sam Hokin wrote:
Caldarale, Charles R wrote:
I'll poke around in the webapp classloader to see if I can find
anything interesting, but in the meantime, is there a /net directory
on the problematic server? If there is and it targets a remote file
system, that might explain the long delay on
From: Jorge Medina [mailto:jmed...@e-dialog.com]
Subject: RE: very slow class loading on initial JSP/servlet
request after restart
Just for curiosity, Is / included in your classpath ?
Note that the classpath of interest is the one JDT uses, not any global one or
any associated
Caldarale, Charles R wrote:
Once you have your test Tomcat set up on the problematic server, try adding
this line to the conf/logging.properties file:
org.apache.jasper.level = ALL
That will display a *lot* of information about what's going on during JSP
compilation. One thing in particular
Sam Hokin wrote:
Caldarale, Charles R wrote:
Once you have your test Tomcat set up on the problematic server, try
adding this line to the conf/logging.properties file:
org.apache.jasper.level = ALL
That will display a *lot* of information about what's going on during
JSP compilation. One
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
Mark,
On 2/25/2009 4:42 AM, Mark Thomas wrote:
Ok. To summarise when you include net.ims.jcms.* in your imports the
page compiles quickly but takes ages to respond to the first request.
I wonder if this is related to loading a specific class in
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
Sam,
On 2/26/2009 10:50 AM, Sam Hokin wrote:
Caldarale, Charles R wrote:
I'll poke around in the webapp classloader to see if I can find
anything interesting, but in the meantime, is there a /net directory
on the problematic server? If there is
I'm glad you got it solved Sam.
It was an exciting topic to follow.
I haven't been reading the mailinglist for a long time, but there might
be a good chance you got the longest thread in the mailinglist
history :)
Cheers.
Pieter
On Thu, 2009-02-26 at 10:31 -0600, Sam Hokin wrote:
Caldarale,
Sam Hokin wrote:
Mark Thomas wrote:
Sam Hokin wrote:
Thanks, Chris. I ran e2fsck with the -c option, which runs badblocks,
when I tested it earlier. And I just ran badblocks again - 0 bad blocks
found. I wish I could fix this by simply as swapping out a bad disk
(notwithstanding Andre's
Thanks, Juha. Actually Pieter suggested it a little while ago, and I've been
trying to get some information out of
strace. The best I can do is to put strace in front of the java command
that's inside catalina.sh. That's the command
that shows with ps -ef when Tomcat is running. BUT,
While I am late to this ... Is this an accurate summary?
- Slow re-load on a server
- Server is a production server
- Other servers are OK so it can only be reproduced on production server
- Initial looks at network seem to be no network activity
- There seems to be a hint at File.exists()
Mark Thomas wrote:
Sam Hokin wrote:
Mark Thomas wrote:
Sam Hokin wrote:
[...]
My thinking is if we can reduce the scope of the problem to importing a
single class, we can then separate out that class and reduce the code in
it bit by bit until we have the bare minimum that causes the
Tim Funk wrote:
While I am late to this ... Is this an accurate summary?
- Slow re-load on a server
- Server is a production server
- Other servers are OK so it can only be reproduced on production server
- Initial looks at network seem to be no network activity
- There seems to be a hint at
Pieter Temmerman wrote:
Try this strace command:
strace -f -etrace=\!futex,gettimeofday,clock_gettime -p
PID_OF_TOMCAT_JAVA
where PID_OF_TOMCAT_JAVA is the PID of your Tomcat, duh! :)
Thanks! That certainly did the trick. The first time I ran strace against the Tomcat process, it brought
André Warnier wrote:
Sam, because there is quite a bit of traffic on this issue, and because
the exact description seems to change a bit over time (e.g.
compilation-only or not), could you remake a summary of what you are
sure of so far ?
1. Initial requests to ANY JSP page on a site,
NEW DISCOVERY - using a wildcard package import results in a much longer
response delay!
I've taken Mark's suggestion that I import individual classes rather than importing all with a wildcard to see if a
particular class is slowing down classloading. It turns out that it's the WILDCARD
André Warnier wrote:
What in this directory path is (are) a symlink(s) ?
/home/sites/ims/jcms/webapps/ROOT/WEB-INF/classes/org/apache/jsp/test6_jsp.class
There are no symlinks in that path.
And how is this different from the other 2 systems that work fine ?
It differs in no known way,
Caldarale, Charles R wrote:
The key lines have a 3+ second response time:
2498 15:24:35 stat64(/net/ims/jcms/Object.class, 0x7b6b9c20) = -1 ENOENT (No such
file or directory) 3.020996
Most of the calls return in much less than a millisecond, but there are a few
ENOENT (not all) responses
Mark Thomas wrote:
Sam Hokin wrote:
Thanks, Chris. I ran e2fsck with the -c option, which runs badblocks,
when I tested it earlier. And I just ran badblocks again - 0 bad blocks
found. I wish I could fix this by simply as swapping out a bad disk
(notwithstanding Andre's desire for
So, looks like I'm late to the party, but will crash in nonetheless.
Sam Hokin wrote:
Christopher Schultz wrote:
On 2/19/2009 6:23 PM, Sam Hokin wrote:
The problem, which spontaneously appeared a few days before Christmas on
this one server, is that the initial request of a JSP or servlet
Juha Laiho wrote:
One tool that I haven't yet seen suggested is 'strace', the Linux system
call tracer. This will show all the calls your application makes to the
operating system. As you say the application is mostly idle during the
delay, it is, in one way or another, waiting for some OS
On Feb 24, 2009, at 3:57 PM, Juha Laiho wrote:
So, looks like I'm late to the party, but will crash in nonetheless.
Sam Hokin wrote:
Christopher Schultz wrote:
On 2/19/2009 6:23 PM, Sam Hokin wrote:
The problem, which spontaneously appeared a few days before
Christmas on
this one server,
On Feb 24, 2009, at 4:16 PM, Sam Hokin wrote:
Juha Laiho wrote:
One tool that I haven't yet seen suggested is 'strace', the Linux
system
call tracer. This will show all the calls your application makes to
the
operating system. As you say the application is mostly idle during
the
delay,
János Löbb wrote:
How about just using tcpdump during the long delay and see what the
machine is doing network wise ?
I have, and network timeout was suggested earlier on this thread. There is no TCP activity of particular interest
during the delay. I've hunted for, and not seen, anything
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
Sam,
On 2/24/2009 1:54 PM, Sam Hokin wrote:
Mark Thomas wrote:
Sam Hokin wrote:
Thanks, Chris. I ran e2fsck with the -c option, which runs badblocks,
when I tested it earlier. And I just ran badblocks again - 0 bad blocks
found. I wish I
Christopher Schultz wrote:
Sam,
On 2/20/2009 6:44 PM, Sam Hokin wrote:
Thanks, Chris. I ran e2fsck with the -c option, which runs badblocks,
when I tested it earlier. And I just ran badblocks again - 0 bad blocks
found.
Did you run badblocks on the array, or on an individual disk?
I would
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
Sam,
On 2/20/2009 6:44 PM, Sam Hokin wrote:
Thanks, Chris. I ran e2fsck with the -c option, which runs badblocks,
when I tested it earlier. And I just ran badblocks again - 0 bad blocks
found.
Did you run badblocks on the array, or on an
A summary so far :
- the problem is : on one certain machine, when deploying any JSP which
imports a certain self-made library of classes, that JSP takes an
inordinate amount of time (one minute or more) to compile for a 5-line JSP
- after compilation, the JSP in question responds totally
Caldarale, Charles R wrote:
From: Sam Hokin [mailto:s...@ims.net]
Subject: very slow class loading on initial JSP/servlet
request after restart
The server that's being a problem is a Dell PowerEdge
1950, quad dual core.
Since I could never believe anything bad could be caused by chocolate
From: André Warnier [mailto:a...@ice-sa.com]
From: Sam Hokin [mailto:s...@ims.net]
The server that's being a problem is a Dell PowerEdge
1950, quad dual core.
Since we have 4 processors here (do we, really? I am not so familiar
with the multi-core architecture), can each of these
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
Sam,
On 2/19/2009 6:23 PM, Sam Hokin wrote:
The problem, which spontaneously appeared a few days before Christmas on
this one server, is that the initial request of a JSP or servlet takes
AGES to respond, usually exceeding several minutes.
Christopher Schultz wrote:
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
Sam,
On 2/19/2009 6:23 PM, Sam Hokin wrote:
The problem, which spontaneously appeared a few days before Christmas on
this one server, is that the initial request of a JSP or servlet takes
AGES to respond, usually
On Feb 20, 2009, at 10:25 AM, André Warnier wrote:
Christopher Schultz wrote:
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
Sam,
On 2/19/2009 6:23 PM, Sam Hokin wrote:
The problem, which spontaneously appeared a few days before
Christmas on
this one server, is that the initial request of a
André Warnier wrote:
Christopher Schultz wrote:
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
Sam,
On 2/19/2009 6:23 PM, Sam Hokin wrote:
The problem, which spontaneously appeared a few days before
Christmas on
this one server, is that the initial request of a JSP or servlet takes
AGES
Caldarale, Charles R wrote:
1) What are the servers that don't have a problem?
One is a single AMD Athlon 64 2800+, running the 64-bit Sun JDK (same version).
The other is a dual AMD Athlon MP 2000+, running the 32-bit Sun JDK (same
version).
And the bad server has 8 Intel Xeon E5310 CPUs
Ron McNulty wrote:
Hi Sam
You jar - does it include any JSP tag files written as JSP fragments?
There are known issues in this area. The dependency management seems to
get confused, resulting in lots of unnecessary compilation of the .tag
files
No, it doesn't, Ron. But thanks for the
Christopher Schultz wrote:
Sam,
On 2/19/2009 6:23 PM, Sam Hokin wrote:
The problem, which spontaneously appeared a few days before Christmas on
this one server, is that the initial request of a JSP or servlet takes
AGES to respond, usually exceeding several minutes.
Good... then you'll have
Caldarale, Charles R wrote:
Run jstack several times during the delay, and then let's compare the set of
thread dumps.
What is the CPU utilization during the delay?
OK, done. The CPU utilization during the delay is tiny, 1% or less. I've run jstack before, during, and after I've
done a
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
Sam,
On 2/20/2009 12:11 PM, Sam Hokin wrote:
jstack.0.txt ran BEFORE my test request.
Your server is completely idle, here.
jstack.1-11.txt ran DURING the long wait.
Observing jstack.1.txt, it looks like this is the only running thread
(complete
Christopher Schultz wrote:
jstack.8.txt is the last thread dump where File.exists was stalled.
http://ims.net/jstack/jstack.9.txt
http://ims.net/jstack/jstack.10.txt
http://ims.net/jstack/jstack.11.txt
The server appears to be idle, here.
It's a little weird that thread 9770 has NO STACK
Sam Hokin wrote:
Christopher Schultz wrote:
jstack.8.txt is the last thread dump where File.exists was stalled.
http://ims.net/jstack/jstack.9.txt
http://ims.net/jstack/jstack.10.txt
http://ims.net/jstack/jstack.11.txt
The server appears to be idle, here.
It's a little weird that thread
I'd like to correct an error I made in my earlier report of this problem. It turns out that the response delay in my
simple test JSP IS during compilation! I ran stat on the files in question, as well as looked at the time shown on the
response page (which is all the test JSP does). (I think
Sam Hokin wrote:
I'd like to correct an error I made in my earlier report of this
problem. It turns out that the response delay in my simple test JSP IS
during compilation!
Ha !
If it may help, I remember seeing an earlier thread this week, I believe
answered by Rainer Jung, providing
Chuck's comment about JSP to Java translation done by Jasper reminded me that I should have included the time on the
Java source file created during JSP compilation. It shows that the Java source file was created immediately, but
accessed again six seconds later (!), presumably by the compiler.
An update. I've written a simple standalone Java routine that calls File.exists(). It never hangs. In fact, I ran it
against the class file that Tomcat creates from a test JSP; it doesn't hang, returns false for the (long) time it takes
Tomcat to create the class file on the slow system, then
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
Sam and David,
On 2/20/2009 1:36 PM, David Smith wrote:
Sam Hokin wrote:
It's a logical RAID array on four physical disks. I've
run fsck on it (when this all first happened) and it's fine.
It _could_ be a disk issue. I've seen disks that scan
Christopher Schultz wrote:
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
Sam and David,
On 2/20/2009 1:36 PM, David Smith wrote:
Sam Hokin wrote:
It's a logical RAID array on four physical disks. I've
run fsck on it (when this all first happened) and it's fine.
It _could_ be a disk issue.
Sam Hokin wrote:
I'd like to correct an error I made in my earlier report of this
problem. It turns out that the response delay in my simple test JSP IS
during compilation! I ran stat on the files in question, as well as
looked at the time shown on the response page (which is all the test
Thanks, Chris. I ran e2fsck with the -c option, which runs badblocks, when I tested it earlier. And I just ran
badblocks again - 0 bad blocks found. I wish I could fix this by simply as swapping out a bad disk (notwithstanding
Andre's desire for intellectual pursuits), but I really think it's
André Warnier wrote:
Sam Hokin wrote:
I'd like to correct an error I made in my earlier report of this
problem. It turns out that the response delay in my simple test JSP
IS during compilation! I ran stat on the files in question, as well
as looked at the time shown on the response page
Sam Hokin wrote:
Thanks, Chris. I ran e2fsck with the -c option, which runs badblocks,
when I tested it earlier. And I just ran badblocks again - 0 bad blocks
found. I wish I could fix this by simply as swapping out a bad disk
(notwithstanding Andre's desire for intellectual pursuits), but
endorse content contained within this transmission.
Date: Fri, 20 Feb 2009 23:50:21 +
From: ma...@apache.org
To: users@tomcat.apache.org
Subject: Re: very slow class loading on initial JSP/servlet request after
restart
Sam Hokin wrote:
Thanks, Chris. I ran e2fsck with the -c option
I've been struggling with a very severe problem on one of my three Tomcat servers since late December. I'm completely
stumped. I'm running 6.0.18 on all three servers, which are all running Fedora Core 9, all currently running Sun's JDK
1.6.0_12. The only difference is hardware. The server
Sam Hokin wrote:
[...]
Not being a Tomcat or Java specialist at all, but this being fairly late
in the day, I am piqued by your nicely told story and just trying to
imagine what could be the cause of the issue you describe. We Belgians,
like Hercule Poirot, have a knack for such things.
From: Sam Hokin [mailto:s...@ims.net]
Subject: very slow class loading on initial JSP/servlet
request after restart
The server that's being a problem is a Dell PowerEdge
1950, quad dual core.
Since I could never believe anything bad could be caused by chocolate
(especially dark chocolate
...@ims.net
To: users@tomcat.apache.org
Sent: Friday, February 20, 2009 12:23 PM
Subject: very slow class loading on initial JSP/servlet request after
restart
I've been struggling with a very severe problem on one of my three Tomcat
servers since late December. I'm completely stumped. I'm running
57 matches
Mail list logo
