Re: WebSphere Question

[Gary Green]

Thank You Tim!

I'll be passing this along in about 30 seconds. 


Gary Green
While the big event is over, there is still time for you to help me with my
efforts to assist those stricken with cancer!
Please support my efforts by visiting:
http://www.active.com/donate/tntsonj/tntsonjGGreen 
 
Thank you.

-Original Message-
From: IBM Mainframe Discussion List [mailto:[EMAIL PROTECTED] On Behalf
Of Timothy Sipples
Sent: Friday, July 11, 2008 10:38 AM
To: IBM-MAIN@BAMA.UA.EDU
Subject: Re: WebSphere Question

You mean WebSphere Application Server for z/OS, right?

It *could* be WAS V6.0x on z/OS 1.4 -- that is technically possible. WAS
V6.1x requires a higher z/OS release level, though (1.7 minimum I think).

OK, a few suggestions:

1. It sounds like there's a strong suspicion there's an application problem.
Starting with a free solution, I'd recommend getting this tool
pronto:

http://www.alphaworks.ibm.com/tech/jinsightlive

It'll help figure out what's going on in the application. If you want to add
a commercial tool to your kit bag -- and there are good reasons to do that
-- then you have a number of choices. I'm most familiar with IBM's Tivoli
Composite Application Manager (ITCAM), but there are others out there as
well such as CA's Wily Introscope.

2. I'm glad to hear there's a plan to leapfrog up to z/OS 1.9 (via 1.7).
That's because starting in 1.6 you can take advantage of zAAPs. For any
non-trivial amount of Java workload it makes perfect sense to get at least
one zAAP. Also a very good idea to get moving on the DB2 front, because
there are likely performance benefits that accrue with many WAS
applications. For example, you can get some zIIP benefit if you're crossing
LPARs for JDBC access.

3. One basic thing you want to look at is garbage collection. There's a
switch you can throw in WAS to log each garbage collection event, and it's
extremely lightweight so you can turn it on in production. As I recall the
rule of thumb is that if you garbage collect more than every 10 seconds or
so you've got a problem. You can solve that problem by increasing the Java
heap size per servant, increasing the number of servants, fixing the
application (to use fewer/smaller objects and/or avoid "leaking"), or some
combination.

4. Are the servants abending (almost always due to an application abend)?
If so, that'll put demands on the system as it starts new servants to keep
the business running.

5. IBM has published some very good problem determination guides. Here are
some links to check:

http://www.redbooks.ibm.com/abstracts/redp6880.html
http://www.redbooks.ibm.com/abstracts/sg246880.html
http://www.redbooks.ibm.com/abstracts/sg247461.html
http://www.redbooks.ibm.com/abstracts/redp6001.html
http://www.redbooks.ibm.com/abstracts/redp6002.html

Hope that helps get you started.

- - - - -
Timothy Sipples
IBM Consulting Enterprise Software Architect Specializing in Software
Architectures Related to System z Based in Tokyo, Serving IBM Japan and IBM
Asia-Pacific
E-Mail: [EMAIL PROTECTED]
--
For IBM-MAIN subscribe / signoff / archive access instructions, send email
to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO Search the
archives at http://bama.ua.edu/archives/ibm-main.html

No virus found in this incoming message.
Checked by AVG - http://www.avg.com
Version: 8.0.138 / Virus Database: 270.4.7/1546 - Release Date: 7/11/2008
6:47 AM

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO
Search the archives at http://bama.ua.edu/archives/ibm-main.html

Re: WebSphere Question

[Timothy Sipples]

You mean WebSphere Application Server for z/OS, right?

It *could* be WAS V6.0x on z/OS 1.4 -- that is technically possible. WAS
V6.1x requires a higher z/OS release level, though (1.7 minimum I think).

OK, a few suggestions:

1. It sounds like there's a strong suspicion there's an application
problem. Starting with a free solution, I'd recommend getting this tool
pronto:

http://www.alphaworks.ibm.com/tech/jinsightlive

It'll help figure out what's going on in the application. If you want to
add a commercial tool to your kit bag -- and there are good reasons to do
that -- then you have a number of choices. I'm most familiar with IBM's
Tivoli Composite Application Manager (ITCAM), but there are others out
there as well such as CA's Wily Introscope.

2. I'm glad to hear there's a plan to leapfrog up to z/OS 1.9 (via 1.7).
That's because starting in 1.6 you can take advantage of zAAPs. For any
non-trivial amount of Java workload it makes perfect sense to get at least
one zAAP. Also a very good idea to get moving on the DB2 front, because
there are likely performance benefits that accrue with many WAS
applications. For example, you can get some zIIP benefit if you're crossing
LPARs for JDBC access.

3. One basic thing you want to look at is garbage collection. There's a
switch you can throw in WAS to log each garbage collection event, and it's
extremely lightweight so you can turn it on in production. As I recall the
rule of thumb is that if you garbage collect more than every 10 seconds or
so you've got a problem. You can solve that problem by increasing the Java
heap size per servant, increasing the number of servants, fixing the
application (to use fewer/smaller objects and/or avoid "leaking"), or some
combination.

4. Are the servants abending (almost always due to an application abend)?
If so, that'll put demands on the system as it starts new servants to keep
the business running.

5. IBM has published some very good problem determination guides. Here are
some links to check:

http://www.redbooks.ibm.com/abstracts/redp6880.html
http://www.redbooks.ibm.com/abstracts/sg246880.html
http://www.redbooks.ibm.com/abstracts/sg247461.html
http://www.redbooks.ibm.com/abstracts/redp6001.html
http://www.redbooks.ibm.com/abstracts/redp6002.html

Hope that helps get you started.

- - - - -
Timothy Sipples
IBM Consulting Enterprise Software Architect
Specializing in Software Architectures Related to System z
Based in Tokyo, Serving IBM Japan and IBM Asia-Pacific
E-Mail: [EMAIL PROTECTED]
--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO
Search the archives at http://bama.ua.edu/archives/ibm-main.html

Re: WebSphere Question

[Gray, Larry - Larry A]

Assuming you are talking about WAS, it can write SMF 120 records if configured 
to do so.  That would be similar to the CICS SMF 110 records.  That would give 
you an idea of what was running during that timeframe.  I would also suggest 
turning on verbose GC.  If you are having heap issues, that should show up 
there.  Overhead on verbose GC is negligable.  If you have a DB2 monitor, you 
can look at that during that time frame and see if there are any heavy/poor DB2 
apps running.


Larry Gray
Large Systems Engineering
Lowe's Companies
336-658-7944

-Original Message-
From: IBM Mainframe Discussion List [mailto:[EMAIL PROTECTED] On Behalf Of Gary 
Green
Sent: Thursday, July 10, 2008 10:29 PM
To: IBM-MAIN@BAMA.UA.EDU
Subject: WebSphere Question

If someone knows where I should ask this question, please tell me where to go. 
;)

Otherwise...

I was chatting with the manager of the tech support group at another company 
and he was telling me that in his shop they experience a terrible slowdown in 
WebSphere most afternoons; "sucks the life out of the machine" comes to mind.  
They do not have anyone that does performance or capacity work to "run the 
numbers"; and they collect very little SMF data even if they did or knew how to 
"run the numbers".  He thinks it's the application programmers poorly written 
code, probably Java doing some daily end of day stuff.  To make matters worse, 
they are still on 1.4, with plans to make a brief stopover on 1.7 before 
heading off to 1.9.

They are pretty much a vanilla IBM shop with a couple of DB2 V7 production 
regions a couple of production CICS regions and some/few regions each for QA, 
Dev and testing.  I think he said WebSphere was... version 6 if that
makes sense... (I am not conversant in WebSphere).   They are on a ~480 mip
Z9.

I know this is not really much to go on but does anyone have any suggestions I 
could pass along?  Perhaps some insight or suggested diagnostic processes?


Gary Green
While the big event is over, there is still time for you to help me with my 
efforts to assist those stricken with cancer!
Please support my efforts by visiting:
 <http://www.active.com/donate/tntsonj/tntsonjGGreen>
http://www.active.com/donate/tntsonj/tntsonjGGreen

Thank you.


--
For IBM-MAIN subscribe / signoff / archive access instructions, send email to 
[EMAIL PROTECTED] with the message: GET IBM-MAIN INFO Search the archives at 
http://bama.ua.edu/archives/ibm-main.html

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO
Search the archives at http://bama.ua.edu/archives/ibm-main.html

Re: WebSphere Question

[Bryan Klimek]

Also, don't be afraid to engage IBM to help you find the offending piece of
code that is causing the high CPU situation. Just this week I had a
z/WebSphere app server 'suck the life out of the machine'. I took and
address space dump, opened an ETR and uploaded the dump. Within 24 hours IBM
was able to identify the TCB that was consuming the CPU along with the java
stack trace. Gave this to the developer and they were able to see an error
in the program logic.

Bryan

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO
Search the archives at http://bama.ua.edu/archives/ibm-main.html

Re: WebSphere Question

[Gary Green]

Forbid Project...   I believe that was me a few weeks back.  My son got a
copy from Netflix for me and after he watched it at his place, his mother
and I watched it here; on the BIG screen. ;)

Well, those thoughts sound very plausible and logical.  My first thought was
memory leakage but I'm not close enough to the product to say that with any
semblance of experience/knowhow.

I appreciate the comments and will be passing them along in a few.

Thanks.


Gary Green
While the big event is over, there is still time for you to help me with my
efforts to assist those stricken with cancer!
Please support my efforts by visiting:
http://www.active.com/donate/tntsonj/tntsonjGGreen 
 
Thank you.

-Original Message-
From: IBM Mainframe Discussion List [mailto:[EMAIL PROTECTED] On Behalf
Of Patrick Falcone
Sent: Friday, July 11, 2008 12:38 AM
To: IBM-MAIN@BAMA.UA.EDU
Subject: Re: WebSphere Question

Hi Gary,
   
  We had a similar problem back at my last place and *sucks the life out of
the machine* is about right.
   
  Sounds like it could be a storage leak/creep within the heap that is
causing the Java heap to go into more frequent garbage collections,
especially if the heap is lightly allocated - hopefully it's at least 512 MB
or above, as free space diminishes over time to the point where a compaction
of the Java heap may be occurring. The compaction process takes considerable
CPU to complete. I'm not sure of the reporting at this point with version 6
of WAS but turning on verbose GC, garbage collection, may shed some light on
this anomaly and if it is indeed heap space related. BTDTGTS.
   
  Of course this could also be a runaway thread, bad logic, in which case
you'll need some tracing to find the culprit but would lean on the above if
it is happening, like, late afternoons around the same time.
   
  I also wonder if log offloads might be causing this but I can't seem to
remember if this was a cause of significant CPU within WAS. I'm sure we'll
get more elegant responses in the morning.
   
  A bit off topic, sorry, I just got done watching the Forbin Project with
my son, great movie, I forget who recommended it but thanks
  
Gary Green <[EMAIL PROTECTED]> wrote:
  If someone knows where I should ask this question, please tell me where to
go. ;)

Otherwise...

I was chatting with the manager of the tech support group at another company
and he was telling me that in his shop they experience a terrible slowdown
in WebSphere most afternoons; "sucks the life out of the machine" comes to
mind. They do not have anyone that does performance or capacity work to "run
the numbers"; and they collect very little SMF data even if they did or knew
how to "run the numbers". He thinks it's the application programmers poorly
written code, probably Java doing some daily end of day stuff. To make
matters worse, they are still on 1.4, with plans to make a brief stopover on
1.7 before heading off to 1.9.

They are pretty much a vanilla IBM shop with a couple of DB2 V7 production
regions a couple of production CICS regions and some/few regions each for
QA, Dev and testing. I think he said WebSphere was... version 6 if that
makes sense... (I am not conversant in WebSphere). They are on a ~480 mip
Z9.

I know this is not really much to go on but does anyone have any suggestions
I could pass along? Perhaps some insight or suggested diagnostic processes?


Gary Green
While the big event is over, there is still time for you to help me with my
efforts to assist those stricken with cancer!
Please support my efforts by visiting:

http://www.active.com/donate/tntsonj/tntsonjGGreen 

Thank you.


--
For IBM-MAIN subscribe / signoff / archive access instructions, send email
to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO Search the
archives at http://bama.ua.edu/archives/ibm-main.html

No virus found in this incoming message.
Checked by AVG - http://www.avg.com
Version: 8.0.138 / Virus Database: 270.4.7/1545 - Release Date: 7/10/2008
6:43 PM

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO
Search the archives at http://bama.ua.edu/archives/ibm-main.html

Re: WebSphere Question

[Gary Green]

Well, "I" know very little about the environment, but I will send an email
asking for more details.

Thanks. 


Gary Green
While the big event is over, there is still time for you to help me with my
efforts to assist those stricken with cancer!
Please support my efforts by visiting:
http://www.active.com/donate/tntsonj/tntsonjGGreen 
 
Thank you.

-Original Message-
From: IBM Mainframe Discussion List [mailto:[EMAIL PROTECTED] On Behalf
Of Chauhan, Jasbir
Sent: Friday, July 11, 2008 7:46 AM
To: IBM-MAIN@BAMA.UA.EDU
Subject: Re: WebSphere Question

Gary,
 
WebSphere is a pretty broad term. There are a bunch of products that run
under the brand name of WebSphere. From the type of workload (java) you are
describing, it appears you maybe talking about Application Server.
Could you elaborate a little on the actual product and its version. 

Regards,
Jasbir
 
 
 
-Original Message-
From: IBM Mainframe Discussion List [mailto:[EMAIL PROTECTED] On Behalf
Of <>
 

--
For IBM-MAIN subscribe / signoff / archive access instructions, send email
to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO Search the
archives at http://bama.ua.edu/archives/ibm-main.html

No virus found in this incoming message.
Checked by AVG - http://www.avg.com
Version: 8.0.138 / Virus Database: 270.4.7/1545 - Release Date: 7/10/2008
6:43 PM

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO
Search the archives at http://bama.ua.edu/archives/ibm-main.html

Re: WebSphere Question

[Chauhan, Jasbir]

Gary,
 
WebSphere is a pretty broad term. There are a bunch of products that run
under the brand name of WebSphere. From the type of workload (java) you
are describing, it appears you maybe talking about Application Server.
Could you elaborate a little on the actual product and its version. 

Regards,
Jasbir
 
 
 
-Original Message-
From: IBM Mainframe Discussion List [mailto:[EMAIL PROTECTED] On
Behalf Of <>
 

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO
Search the archives at http://bama.ua.edu/archives/ibm-main.html

Re: WebSphere Question

[Patrick Falcone]

Hi Gary,
   
  We had a similar problem back at my last place and *sucks the life out of the 
machine* is about right.
   
  Sounds like it could be a storage leak/creep within the heap that is causing 
the Java heap to go into more frequent garbage collections, especially if the 
heap is lightly allocated - hopefully it's at least 512 MB or above, as free 
space diminishes over time to the point where a compaction of the Java heap may 
be occurring. The compaction process takes considerable CPU to complete. I'm 
not sure of the reporting at this point with version 6 of WAS but turning on 
verbose GC, garbage collection, may shed some light on this anomaly and if it 
is indeed heap space related. BTDTGTS.
   
  Of course this could also be a runaway thread, bad logic, in which case 
you'll need some tracing to find the culprit but would lean on the above if it 
is happening, like, late afternoons around the same time.
   
  I also wonder if log offloads might be causing this but I can't seem to 
remember if this was a cause of significant CPU within WAS. I'm sure we'll get 
more elegant responses in the morning.
   
  A bit off topic, sorry, I just got done watching the Forbin Project with my 
son, great movie, I forget who recommended it but thanks
  
Gary Green <[EMAIL PROTECTED]> wrote:
  If someone knows where I should ask this question, please tell me where to
go. ;)

Otherwise...

I was chatting with the manager of the tech support group at another company
and he was telling me that in his shop they experience a terrible slowdown
in WebSphere most afternoons; "sucks the life out of the machine" comes to
mind. They do not have anyone that does performance or capacity work to
"run the numbers"; and they collect very little SMF data even if they did or
knew how to "run the numbers". He thinks it's the application programmers
poorly written code, probably Java doing some daily end of day stuff. To
make matters worse, they are still on 1.4, with plans to make a brief
stopover on 1.7 before heading off to 1.9.

They are pretty much a vanilla IBM shop with a couple of DB2 V7 production
regions a couple of production CICS regions and some/few regions each for
QA, Dev and testing. I think he said WebSphere was... version 6 if that
makes sense... (I am not conversant in WebSphere). They are on a ~480 mip
Z9.

I know this is not really much to go on but does anyone have any suggestions
I could pass along? Perhaps some insight or suggested diagnostic processes?


Gary Green
While the big event is over, there is still time for you to help me with my
efforts to assist those stricken with cancer!
Please support my efforts by visiting:

http://www.active.com/donate/tntsonj/tntsonjGGreen 

Thank you.


--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO
Search the archives at http://bama.ua.edu/archives/ibm-main.html

WebSphere Question

[Gary Green]

If someone knows where I should ask this question, please tell me where to
go. ;)
 
Otherwise...
 
I was chatting with the manager of the tech support group at another company
and he was telling me that in his shop they experience a terrible slowdown
in WebSphere most afternoons; "sucks the life out of the machine" comes to
mind.  They do not have anyone that does performance or capacity work to
"run the numbers"; and they collect very little SMF data even if they did or
knew how to "run the numbers".  He thinks it's the application programmers
poorly written code, probably Java doing some daily end of day stuff.  To
make matters worse, they are still on 1.4, with plans to make a brief
stopover on 1.7 before heading off to 1.9.
 
They are pretty much a vanilla IBM shop with a couple of DB2 V7 production
regions a couple of production CICS regions and some/few regions each for
QA, Dev and testing.  I think he said WebSphere was... version 6 if that
makes sense... (I am not conversant in WebSphere).   They are on a ~480 mip
Z9.
 
I know this is not really much to go on but does anyone have any suggestions
I could pass along?  Perhaps some insight or suggested diagnostic processes?

 
Gary Green
While the big event is over, there is still time for you to help me with my
efforts to assist those stricken with cancer!
Please support my efforts by visiting:
 
http://www.active.com/donate/tntsonj/tntsonjGGreen 
 
Thank you.
 

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO
Search the archives at http://bama.ua.edu/archives/ibm-main.html