I will add our own recent experience to this. We have an application that
does not play well with others, for whatever reason. Could be the other
apps, we don't know. What we DO know is that you need to probably turn on
verbose garbage collection in the WAS admin panel, and then look at how
often you are doing gargbage collection by examining the nativestderr.log
file in the server tasks directory (e.g.
/opt/WebSphere/AppServer/logs/server{x})
You want to see garbage collections like this:
<AF[3]: Allocation Failure. need 528 bytes, 94867 ms since last AF>
<AF[3]: managing allocation failure, action=1 (0/338688584)
(17825720/17825720)>
<GC(3): GC cycle started Wed Jun 16 12:55:12 2004
<GC(3): freed 294419184 bytes, 87% free (312244904/356514304), in 224 ms>
<GC(3): mark: 185 ms, sweep: 39 ms, compact: 0 ms>
<GC(3): refs: soft 0 (age >= 32), weak 35, final 1577, phantom 0>
<AF[3]: completed in 226 ms>
Here it was 94 seconds since it did a garbage collection. Albeit, this was
only the 3rd one since the server came up, but it wasn't thrashing itself
to death trying to find enough memory. You want a few seconds between
collections if you can get them.
You can see then that the java heap was 356,514,304 bytes, and that it
freed 294,419,184 in 226 ms.
We started seeing this with the badly behaved app not in its own server:
<AF[203]: Allocation Failure. need 16400 bytes, 146 ms since last AF>
<AF[203]: managing allocation failure, action=2 (230994552/356514304)>
<GC(203): GC cycle started Wed Jun 16 15:16:47 2004
<GC(203): freed 25982808 bytes, 72% free (256977360/356514304), in 364
ms>
<GC(203): mark: 296 ms, sweep: 68 ms, compact: 0 ms>
<GC(203): refs: soft 0 (age >= 32), weak 0, final 5, phantom 0>
<AF[203]: completed in 365 ms>
<AF[204]: Allocation Failure. need 16400 bytes, 135 ms since last AF>
<AF[204]: managing allocation failure, action=2 (230766720/356514304)>
<GC(204): GC cycle started Wed Jun 16 15:16:48 2004
<GC(204): freed 26097792 bytes, 72% free (256864512/356514304), in 373
ms>
<GC(204): mark: 303 ms, sweep: 70 ms, compact: 0 ms>
<GC(204): refs: soft 0 (age >= 32), weak 0, final 1, phantom 0>
<AF[204]: completed in 374 ms>
<AF[205]: Allocation Failure. need 16400 bytes, 577 ms since last AF>
<AF[205]: managing allocation failure, action=2 (223579440/356514304)>
<GC(205): GC cycle started Wed Jun 16 15:16:49 2004
<GC(205): freed 33351448 bytes, 72% free (256930888/356514304), in 715
ms>
<GC(205): mark: 592 ms, sweep: 123 ms, compact: 0 ms>
<GC(205): refs: soft 0 (age >= 32), weak 1, final 19, phantom 0>
<AF[205]: completed in 772 ms>
<AF[206]: Allocation Failure. need 16400 bytes, 136 ms since last AF>
<AF[206]: managing allocation failure, action=2 (231656440/356514304)>
<GC(206): GC cycle started Wed Jun 16 15:16:50 2004
<GC(206): freed 25312584 bytes, 72% free (256969024/356514304), in 494
ms>
<GC(206): mark: 425 ms, sweep: 69 ms, compact: 0 ms>
<GC(206): refs: soft 3 (age >= 32), weak 0, final 1, phantom 0>
<AF[206]: completed in 513 ms>
AF[203] happened 146 microseconds after AF[202] and took 365 microseconds
to finish.
AF[204] happened 135 microseconds after AF[203] and took 374 microseconds
to finish.
AF[205] happened 577 microseconds after AF[204] and took 772 microseconds
to finish.
AF[206] happened 136 microseconds after AF[205] and took 513 microseconds
to finish.
You can deduce from this that was had huge problem because we were spending
twice as much time in garbage collection than we were in execution. The
time spend in garbage collection is going to send CP through the ceiling.
Secondly we found that for whatever reason the JVM was not compacting
itself. You can set the JVM to patially compact a 'segment' of the heap
each garbage collection and this helped us by giving a better GC to GC
interval, but the collections took longer becase we were forcing it to
compact.
We finally isolated the application in it's own server, where compaction
happens when it needs to, and it generally behaves better.
I hope this gives you a little bit of an idea where to look for some
bottlenecks in your working set (real memory) and your JVM heap size
-J
David Boyes
<[EMAIL PROTECTED]
e.net> To
Sent by: Linux on [EMAIL PROTECTED]
390 Port cc
<[EMAIL PROTECTED]
IST.EDU> Subject
Re: J2EE performance?
06/18/2004 02:29
PM
Please respond to
Linux on 390 Port
<[EMAIL PROTECTED]
IST.EDU>
> > * z800
> > * 2 IFL's (tested using single and dual, still slow) in LPAR
> > * z/VM 4.4.0
> > * guest lan networking
> > * 256 meg RAM
> > * 512 meg vdisk swap
That's a bit on the anemic side for Java apps.
> > * Turning of hz_timer patch helps lower CPU utilization footprint of
> > the java processes, but of course, then the system never sleeps.
Yep.
> > * When running a J2EE app, it takes often over a minute to finish
> > rendering a page to a browser.
Yep. Most J2EE apps are pretty inefficient in their use of resources.
> > * When running a J2EE app, java consumes an entire IFL of
> cpu if we let
> > it.
Nothing surprising here -- the same thing happens on Intel boxes; nobody
notices because they're usually not shared.
> > 1. Is J2EE a cpu resource hog for you?
Yes. It's usually caused by careless application programs assuming
resources are infinite and expecting the system deployer to add hardware
until the poor app can be supported. The JVM version can make a dramatic
difference in how piggy the app is (we saw significant differences from
1.3.x to 1.4.x, both up and down depending what Java features the apps
used).
> > 2. Are you running Linux under VM or right on the LPAR?
Under VM. The cost case for LPAR is very difficult (if not impossible)
to make.
> > 3. Do you charge users for MIP usage? If so, how does
> this high CPU
> > utilization not eat your customer alive in billed costs?
Not for apps involving Java. We measure it for capacity planning
purposes, but don't try to bill it.
> > 4. Does anyone have some advice on how to run J2EE services more
> > efficiently, faster and easier on cpu?
Add resources to the 390 (either additional IFLs or convert to a 890/990
CPU), or segment the app such that the computationally-intensive
components are not run on the 390. 390 processors aren't designed for
problems that assume that CPU is a free resource.
> > 5. Which J2EE server software for mainframe Linux do you prefer and
> > why?
JBoss is the least piggy of the bunch, but both Websphere and BEA run
fairly well given enough resources. The configuration you listed above
is a little light for either one.
It's usually not the server software that causes the problems; it's more
likely to be the application itself if it executes on the server side.
> > 6. Lastly, why would you choose to run J2EE on mainframe Linux as
> > apposed to some stand-alone system (Unix, Windows, Other)?
Convenience, proximity to the data, ease of creation for testing
environments.
----------------------------------------------------------------------
For LINUX-390 subscribe / signoff / archive access instructions,
send email to [EMAIL PROTECTED] with the message: INFO LINUX-390 or
visit
http://www.marist.edu/htbin/wlvindex?LINUX-390
----------------------------------------------------------------------
For LINUX-390 subscribe / signoff / archive access instructions,
send email to [EMAIL PROTECTED] with the message: INFO LINUX-390 or visit
http://www.marist.edu/htbin/wlvindex?LINUX-390
