Thanks for a comprehensive statement of the problem - so many people don't
include the basics, let alone the details!
A few thoughts inline.
On 16 March 2010 13:58, Patrik Kudo <k...@pingpong.net> wrote:
> We run a load test scenario using the Proxysniffer load testing tool on a
> machine connected to the same switch as the server under load. The load test
> simulates 3100 users
Why this number? What happens if you increase it - does the incidence of
the problem increase? This might make it easier to track down.
> looping over 27 pages of varying complexity.
Again, can you force the issue by tuning which pages are requested?
> Proxysniffer reports all errors as "Network Connection aborted by Server"
> but when we look at each error in detail we can see that they don't all
> occur at the same stage in the request cycle. Some occur on "transmit http
> request", some on "open network connection", some on "wait for server
> response", but all within the same second.
>
It'd be interesting to run (say) Wireshark and sniff the TCP connections.
In particular, that sounds like TCP RSTs coming off the server but it would
be good to verify that and to see at which points in the negotiation they
happen.
>
> The application logs show nothing unusual. The access logs show nothing
> unusual. We've included the session ids in the tomcat logs and the failing
> urls doesn't show up in the access log at all for the given session id
> (cookies are shown in the error report).
>
That's interesting; I'll leave better-qualified people to comment on what
code paths this eliminates.
> We've been able to match some of the errors with full collections reported
> by the flags "-verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps" but
> some errors occur where there are no full GC occuring.
>
How long do your full GCs take?
> # Memory limits (we've tried both higher and lower values here)
> JAVA_OPTS="${JAVA_OPTS} -XX:MaxPermSize=192m -Xmx1800m -Xms1800m"
>
That's a small part of a 32G machine, but you're seeing no out of memory
errors so it's a sign of good design and coding ;-).
FreeBSD 8.0-RELEASE-p2 with diablo-jdk1.6.0 (we also tried openjdk6).
Can you tell us *exactly* which versions of the JDKs?
http://www.freebsd.org/java/ tells me 1.6.0-7 is current - sorry, I'm not as
well up on FreeBSD Java versions as some other OSs.
- Peter
