Re: jk Status not showing errors

2009-06-02 Thread Matthew Laird
Unfortunately I'm not seeing that.  What I did was start both Tomcats in 
my LB pair, start Apache, then I take the second Tomcat down to see if 
it will detect it being failed.


Unfortunately it never seems to, it just shows the second as OK/IDLE, 
and happily directs all requests to the first.  This concerns me, 
because if the second were to fail, then later the first, everything 
would die and I'd have no advance warning.  I can't seem to make it ping 
and detect a dead Tomcat.


I am using the latest version of mod_jk, I upgraded that before I began 
playing with the load balancer settings.  I'd appreciate any feedback on 
what I might be doing wrong.  Thanks.


workers.properties:

worker.list=production,development,old,jkstatus

worker.production.type=lb
worker.production.balance_workers=production1,production2
worker.production.sticky_session=True
worker.production.method=S

worker.lbbasic.type=ajp13
worker.lbbasic.connect_timeout=1
worker.lbbasic.recovery_options=7
worker.lbbasic.socket_keepalive=1
worker.lbbasic.socket_timeout=60
worker.lbbasic.ping_mode=CI

worker.production1.reference=worker.lbbasic
worker.production1.port=8009
worker.production1.host=localhost

worker.production2.reference=worker.lbbasic
worker.production2.port=8012
worker.production2.host=localhost

worker.development.port=8010
worker.development.host=localhost
worker.development.type=ajp13

worker.old.port=8011
worker.old.host=localhost
worker.old.type=ajp13

worker.jkstatus.type=status


Lawrence Lamprecht wrote:

I do not know if this is relevant or not, but I have just installed the latest 
version of mod_jk and the jkstatus is very much better than it used to be.

I had the same issue with loadbalancers not showing when they are offline or 
broken. With the latest version, jksataus has the possibility to auto refresh 
itself. This now shouws when load balancers go down without a request being 
send to it. It is pretty dynamic as well. I ran several tests where I took one 
of the balancers down, and left jkstatus refreshing every 10 seconds and that 
told me that the worker was in error.

It also shows you that the work is OK - IDLE when the worker is not being used 
but is good. As soon as it receives a request the status then changes to OK.

Hope this helps.

Kind regards / Met vriendelijke groet,
Lawrence Lamprecht
Application Content Manager
QUADREM Netherlands B.V.
Kabelweg 61, 1014 BA  Amsterdam
Post Office Box 20672, 1001 NR  Amsterdam
Office: +31 20 880 41 16
Mobile: +31 6 13 14 26 31
Fax: +31 20 880 41 02



Read our blog: Intelligent Supply Management - Your advantage


-Original Message-
From: Rainer Jung [mailto:rainer.j...@kippdata.de] 
Sent: Saturday, May 30, 2009 2:46 PM

To: Tomcat Users List
Subject: Re: jk Status not showing errors

On 29.05.2009 22:50, Matthew Laird wrote:

Good afternoon,

I've been trying to get the jkstatus component of mod_jk running, and
I'm not quite sure what I'm doing wrong in trying to have it report dead
Tomcat instances.

I have two tomcat instances setup in a load balancer, as a test I've
taken down one of them.  However the jkstatus screen still shows both of
them as OK.  I'm not sure what I'm missing from my workers.properties
file to make it test the Tomcat and report a failed instance, so I can
set Nagios to monitor this page and report problems.

My workers.properties is:

worker.list=production,development,old,jkstatus

worker.production.type=lb
worker.production.balance_workers=production1,production2
worker.production.sticky_session=True
worker.production.method=S

worker.lbbasic.type=ajp13
worker.lbbasic.connect_timeout=1
worker.lbbasic.recovery_options=7
worker.lbbasic.socket_keepalive=1
worker.lbbasic.socket_timeout=60

worker.production1.reference=worker.lbbasic
worker.production1.port=8009
worker.production1.host=localhost
#worker.production1.redirect=production2

worker.production2.reference=worker.lbbasic
worker.production2.port=8012
worker.production2.host=localhost
#worker.production2.activation=disabled

worker.development.port=8010
worker.development.host=localhost
worker.development.type=ajp13

worker.old.port=8011
worker.old.host=localhost
worker.old.type=ajp13

worker.jkstatus.type=status


Any advice on extra options to make jkstatus check and report when one
of the Tomcat instances isn't responding would be appreciated.


I assume, that the actual error detection works and you are really only
asking about display in status worker. I also assume your are using a
recent mod_jk. Nevertheless do yourself a favor and look at the Timeouts
documentation page to improve your configuration.

Until recently, only workers used via a load balancing worker had good
manageability with jkstatus. Very recently also pure AJP workers without
any load balancer got more useful information in their display.

So let's talk about your worker production. Whenever a request comes
in the lb first checks whether it already carries a session for one

Re: jk Status not showing errors

2009-06-02 Thread Matthew Laird
I'm not seeing anything like that.  I just took both Tomcats down, I 
instantly get the 503 from Apache when I try to load the application.


However tailing the mod_jk.log, I just see entries like this:

[Tue Jun 02 12:36:23 2009] jkstatus www.innatedb.ca 0.000360
[Tue Jun 02 12:36:26 2009] jkstatus www.innatedb.ca 0.000263
[Tue Jun 02 12:36:39 2009] production www.innatedb.ca 0.498998
[Tue Jun 02 12:36:40 2009] jkstatus www.innatedb.ca 0.000282

mod_jk seems happy sending the requests to Tomcat, and doesn't seem to 
notice there's no actual Tomcat responding.  Only after a few minutes 
does the JK Status screen go to ERR/REC for both.


I would think this is the kind of thing mod_jk should notice instantly, 
when there's no Tomcat where there should be one.


Or am I missing something?

Thanks.

Lawrence Lamprecht wrote:

What you could do is tail -f mod_jk.log file. Then take down the tomcat, see if 
the errors appear. You should see something like the following.

Good Entries to Track

Attempting to map context URI '/search-engine*'
ajp_unmarshal_response::jk_ajp_common.c (621): status = 302
 
Maintaining worker loadbalancer1

Maintaining worker prod_se1
Maintaining worker prod_se2

Maintaining worker prod_sea
Maintaining worker prod_seb

service::jk_lb_worker.c (612): service worker=prod_sea jvm_route=prod_sea
service::jk_lb_worker.c (612): service worker=prod_seb jvm_route=prod_seb
service::jk_lb_worker.c (612): service worker=prod_sea jvm_route=prod_se1
service::jk_lb_worker.c (612): service worker=prod_seb jvm_route=prod_se2

Possible Error Entries

Error connecting to tomcat. Tomcat is probably not started or is listening on 
the wrong port. worker=prod_se1 failed
Error connecting to tomcat. Tomcat is probably not started or is listening on 
the wrong port. worker=prod_se2 failed

You should be able to trace where your config is problematic.

Kind regards / Met vriendelijke groet,
Lawrence Lamprecht

-Original Message-
From: Matthew Laird [mailto:lai...@sfu.ca] 
Sent: Tuesday, June 02, 2009 8:53 PM

To: Tomcat Users List
Subject: Re: jk Status not showing errors

Unfortunately I'm not seeing that.  What I did was start both Tomcats in 
my LB pair, start Apache, then I take the second Tomcat down to see if 
it will detect it being failed.


Unfortunately it never seems to, it just shows the second as OK/IDLE, 
and happily directs all requests to the first.  This concerns me, 
because if the second were to fail, then later the first, everything 
would die and I'd have no advance warning.  I can't seem to make it ping 
and detect a dead Tomcat.


I am using the latest version of mod_jk, I upgraded that before I began 
playing with the load balancer settings.  I'd appreciate any feedback on 
what I might be doing wrong.  Thanks.


workers.properties:

worker.list=production,development,old,jkstatus

worker.production.type=lb
worker.production.balance_workers=production1,production2
worker.production.sticky_session=True
worker.production.method=S

worker.lbbasic.type=ajp13
worker.lbbasic.connect_timeout=1
worker.lbbasic.recovery_options=7
worker.lbbasic.socket_keepalive=1
worker.lbbasic.socket_timeout=60
worker.lbbasic.ping_mode=CI

worker.production1.reference=worker.lbbasic
worker.production1.port=8009
worker.production1.host=localhost

worker.production2.reference=worker.lbbasic
worker.production2.port=8012
worker.production2.host=localhost

worker.development.port=8010
worker.development.host=localhost
worker.development.type=ajp13

worker.old.port=8011
worker.old.host=localhost
worker.old.type=ajp13

worker.jkstatus.type=status


Lawrence Lamprecht wrote:

I do not know if this is relevant or not, but I have just installed the latest 
version of mod_jk and the jkstatus is very much better than it used to be.

I had the same issue with loadbalancers not showing when they are offline or 
broken. With the latest version, jksataus has the possibility to auto refresh 
itself. This now shouws when load balancers go down without a request being 
send to it. It is pretty dynamic as well. I ran several tests where I took one 
of the balancers down, and left jkstatus refreshing every 10 seconds and that 
told me that the worker was in error.

It also shows you that the work is OK - IDLE when the worker is not being used 
but is good. As soon as it receives a request the status then changes to OK.

Hope this helps.

Kind regards / Met vriendelijke groet,
Lawrence Lamprecht
Application Content Manager
QUADREM Netherlands B.V.
Kabelweg 61, 1014 BA  Amsterdam
Post Office Box 20672, 1001 NR  Amsterdam
Office: +31 20 880 41 16
Mobile: +31 6 13 14 26 31
Fax: +31 20 880 41 02



Read our blog: Intelligent Supply Management - Your advantage


-Original Message-
From: Rainer Jung [mailto:rainer.j...@kippdata.de] 
Sent: Saturday, May 30, 2009 2:46 PM

To: Tomcat Users List
Subject: Re: jk Status not showing errors

On 29.05.2009 22:50, Matthew Laird wrote:

Good afternoon,

I've been trying to get

Re: jk Status not showing errors

2009-06-02 Thread Matthew Laird


Rainer Jung wrote:


Assuming that you did refresh the jkstatus display: what is your test
client? The fact that you see OK/IDLE, but all requests go to the other
node indicates, that you are using requests with associated session, so
the balancer is not allowed to send them to the other node and thus does
not detect the down node. Check to remove the JSESSIONID cookie before
sending requests, or use a client which allows cookie disabling (like curl).


Is there any way to make it ping and detect a dead Tomcat without a 
request coming in?  I thought I was doing that with the 
worker.lbbasic.ping_mode=CI setting.


Thanks.

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org



jk Status not showing errors

2009-05-29 Thread Matthew Laird

Good afternoon,

I've been trying to get the jkstatus component of mod_jk running, and 
I'm not quite sure what I'm doing wrong in trying to have it report dead 
Tomcat instances.


I have two tomcat instances setup in a load balancer, as a test I've 
taken down one of them.  However the jkstatus screen still shows both of 
them as OK.  I'm not sure what I'm missing from my workers.properties 
file to make it test the Tomcat and report a failed instance, so I can 
set Nagios to monitor this page and report problems.


My workers.properties is:

worker.list=production,development,old,jkstatus

worker.production.type=lb
worker.production.balance_workers=production1,production2
worker.production.sticky_session=True
worker.production.method=S

worker.lbbasic.type=ajp13
worker.lbbasic.connect_timeout=1
worker.lbbasic.recovery_options=7
worker.lbbasic.socket_keepalive=1
worker.lbbasic.socket_timeout=60

worker.production1.reference=worker.lbbasic
worker.production1.port=8009
worker.production1.host=localhost
#worker.production1.redirect=production2

worker.production2.reference=worker.lbbasic
worker.production2.port=8012
worker.production2.host=localhost
#worker.production2.activation=disabled

worker.development.port=8010
worker.development.host=localhost
worker.development.type=ajp13

worker.old.port=8011
worker.old.host=localhost
worker.old.type=ajp13

worker.jkstatus.type=status


Any advice on extra options to make jkstatus check and report when one 
of the Tomcat instances isn't responding would be appreciated.


Thanks.

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org



Tomcat not using multiple cores

2008-10-16 Thread Matthew Laird

We're pulling our hair out with a Tomcat issue.

We have an in-house application running on Tomcat 5.5 with Sun JDK 1.6.
 The machine is an x86 dual-CPU, quad core (8 cores total) with 16GB of
RAM.  We're running OpenSuSE 10.2, 32-bit.  Java memory size set to 2GB,
multi-threaded GC enabled.

What occurs is when a user clicks a certain kind of analysis on the
website, data is retrieved from a database and then a lot of formatting
is done before returning it to the user.  This typically causes 100% CPU
usage for this thread for a few minutes (bioinformatics application,
that part isn't going to change).

Unfortunately what then occurs is all other threads suddenly become
unusably slow.  The entire web application grinds to a halt until this
thread that's running hot completes.

Looking at top, it appears that these threads aren't spreading among all
the cores.  I see one core go to 100% usage, and the others stay at 100%
idle.  So we're running multi-thread, but because everything is staying
on the same core, we're still getting thread contention that's bringing
the entire application to its knees.

The only time I began to see the other cores actually start being used
is when I enabled multi-threaded GC.  But that doesn't give much
improvement since the threads responding the web requests are still all
on the same core.

I'm not sure how to convince the Tomcat/Java container to spread its
threads among the cores.

Thanks.

--
Matthew Laird
Lead Software Developer, Bioinformatics
Brinkman Laboratory, MBB Dept.
Simon Fraser University


-
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]



Re: Tomcat not using multiple cores

2008-10-16 Thread Matthew Laird

From the OS, no.

From Tomcat, as far as I understand you can only do 2GB per Tomcat 
instance.  Please correct me if I'm wrong.


Jim Cox wrote:

On Thu, Oct 16, 2008 at 10:30 PM, Matthew Laird [EMAIL PROTECTED] wrote:
[...lines snipped...]

We have an in-house application running on Tomcat 5.5 with Sun JDK 1.6.
 The machine is an x86 dual-CPU, quad core (8 cores total) with 16GB of
RAM.  We're running OpenSuSE 10.2, 32-bit.  Java memory size set to 2GB,
multi-threaded GC enabled.

[...rest of post snipped...]

Apologizing in advance for straying a off-topic, but have you had any issues
seeing the full 16GB with a 32-bit Linux install?


-
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]



Re: Tomcat not using multiple cores

2008-10-16 Thread Matthew Laird


Caldarale, Charles R wrote:


The only time I began to see the other cores actually start being used is when I 
enabled multi-threaded GC.  But that doesn't give much improvement since the threads 
responding the web requests are still all on the same core.

The most likely cause is internal synchronization in the webapp or the database 
it references.


I've heard similar from someone else but I'm not sure how that's 
possible.  The app is not at all threaded, is 100% read-only from the 
database (aside from creation of temp tables which have no interaction 
between client requests) and the MySQL server is on another machine 
humming along without any bottlenecks that I can see.


But of course I'm not a Java or JVM expert and have no idea what kind of 
interlinks can exist between different client connections.  But to my 
knowledge they're all pretty straight forward handlers, get data from 
database, format, return to user


I'm leaning more towards GC issues.  I setup Tomcat on a 64-bit machine 
and tried a few configurations:


-Xms4096M -Xmx4096M -server -XX:+DisableExplicitGC -XX:+UseConcMarkSweepGC

-Xms4096M -Xmx4096M -server

-Xms2048M -Xmx2048M -server

As I went through each of those the app became more and more sluggish 
and a single core finally in the last configuration did this 100% CPU 
usage again.


Now, two issues I see.  First, it's not being aggressive enough at 
spreading the load among different cores.  Second, GC does seem to be an 
issue.


Unless I'm missing something, which I might be.

Thanks.

-
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]