Re: blocked / hanging on call to HandlerRequest.checkRequest
Hi Bill If you just want the problem to go away, then look for the attribute request.registerRequests in http://tomcat.apache.org/tomcat-5.5-doc/config/ajp.html . If that is set to false, then there is no locking within the checkRequest method. You lose that ability to get stats for the request threads via JMX (including the manager status page), but that method also quits being a bottleneck if you get a flood of new requests. Thanks for the reply, I found that option yesterday and set up my Connector element with request.registerRequests=false which solved that, then I got blocking on the JspServletWrapper.service method - so I precompiled the jsps - which solved that, then I got blocking on some of my own code where I'd never had a problem before, in short I think the box is simply overloaded - so I'm going to upgrade it, the 5 minute load average peaks at around 20 and is generally over 6 during the day time. Cheers for your reply, Simon - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
blocked / hanging on call to HandlerRequest.checkRequest
Hi I'm running tomcat 5.5.25 on debian sarge 32 bit linux (2.6.8 kernel) with ~ 1.5GB ram on a pentium 4 2GHz with a mysql db 5.0.27 I've got a configuration with apache mod_jk 1.2.25 balancing to 2 tomcats which are both running running on jdk 1.6.0_16 -Xmx=256M periodically, generally at busy times, looking at the JK Status Manager the busy count on one of the tomcats will go up and the requests channelled through that container will start to hang, the busy count will steadily increase the throughput will drop dramatically (i.e. the Acc column in jk_status will stop incrementing by 30 every 10secs and go down to like 4), This will continue until I either stop that tomcat member through the JK Status Manager - by editing the worker settings or the thread count goes up to over the number of permitted apache requests (at 150 at the moment) and apache is restarted automatically by an out of process monitoring app. If I stop the tomcat instance through the JK Status Manager, then the busy count will gradually (over a period of 5 - 10 mins) decrease and get to 0. I took a thread dump by tee-ing the output of catalina.out and then sending a kill -SIGQUIT pid when it was in the described busy state and lots of threads The crux of that seemed to be a lot of threads blocked / waiting for lock on a lock / monitor held in the HandlerRequest.checkRequest here is the printout of the thread holding the lock which is in the RUNNABLE state: TP-Processor65 daemon prio=10 tid=0x08bc9400 nid=0x54bd runnable [0x55dd] java.lang.Thread.State: RUNNABLE at java.lang.Class.getMethod(Class.java:1605) at org.apache.commons.modeler.BaseModelMBean.setManagedResource(BaseModelMBean.java:764) at org.apache.commons.modeler.ManagedBean.createMBean(ManagedBean.java:393) at org.apache.commons.modeler.Registry.registerComponent(Registry.java:835) at org.apache.jk.common.ChannelSocket.registerRequest(ChannelSocket.java:466) at org.apache.jk.common.HandlerRequest.checkRequest(HandlerRequest.java:357) - locked 0x4490ee38 (a java.lang.Object) at org.apache.jk.common.HandlerRequest.decodeRequest(HandlerRequest.java:367) at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:261) at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:773) at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:703) at org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:895) at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:689) at java.lang.Thread.run(Thread.java:619) then lots of the following types of threads (e.g. 35) all blocked TP-Processor63 daemon prio=10 tid=0x09ddc800 nid=0x549f waiting for monitor entry [0x55d3] java.lang.Thread.State: BLOCKED (on object monitor) at org.apache.jk.common.HandlerRequest.checkRequest(HandlerRequest.java:357) - waiting to lock 0x4490ee38 (a java.lang.Object) at org.apache.jk.common.HandlerRequest.decodeRequest(HandlerRequest.java:367) at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:261) at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:773) at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:703) at org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:895) at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:689) at java.lang.Thread.run(Thread.java:619) TP-Processor62 daemon prio=10 tid=0x09dd4c00 nid=0x549e waiting for monitor entry [0x55ce] java.lang.Thread.State: BLOCKED (on object monitor) at org.apache.jk.common.HandlerRequest.checkRequest(HandlerRequest.java:357) - waiting to lock 0x4490ee38 (a java.lang.Object) at org.apache.jk.common.HandlerRequest.decodeRequest(HandlerRequest.java:367) at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:261) at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:773) at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:703) at org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:895) at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:689) at java.lang.Thread.run(Thread.java:619) .etc here is a typical reading from the JK Server Manager for the for the balanced members NametomcatA Typeajp13 Hostlocalhost:8011 Addr127.0.0.1:8011 Act ACT State OK D 0 F 100 M 1 V 104 Acc 42540 Err 0 CE 426 RE 0 Wr 34M Rd 792M Busy80 Max 92 Route tomcatA RR Cd Cd 0/0 NametomcatB Typeajp13 Hostlocalhost:8012 Addr127.0.0.1:8012 Act ACT State OK D 0 F 100 M 1 V 97 Acc 42719 Err 0 CE 377 RE 0 Wr 39M Rd 807M Busy4 Max 57 Route tomcatB RR Cd Cd 0/0 I'm been
Re: jk load balancing based upon ip address rather than session id
Hi Chris, So just to follow up on your post, So, really, it's not that you want to load balance based upon IP address... you really want to predictably choose a member of the server farm based upon some knowledge of the client such that, regardless of the domain name used, the initial request (and therefore all the rest) go to a particular member. That's exactly it. I want all requests from the same client to be channeled through to the same tomcat instance, regardless of the domain. Sorry for the confusion. You may be able to do this, but you'll certainly have to hack mod_jk in order to do it. I believe there is a method in mod_jk that chooses the jvmRoute for the first (JSESSIONID-less) request. If you were to modify that algorithm, you could achieve your goal, here. I'm a little worried that you might implement a fragile algorithm, though, and end up with an unbalanced load balancer. I'll check this out, I think you could be right, it will probably lead to a lumpy load profile accross the tomcat servers, but I might give it a go. Do you have options other than using SSO? Come to think of it... how does SSO work when you switch domains with even a single server -- that is, without load balancing in the mix? I would expect that, since you are switching domains, your browser would not send a JSESSIONID cookie to the server, and thus you would not be recognized as having an existing session. ??! I set a tracking gif on each page pointing to a single domain e.g. tracker.first-domain.com this is served up by the same container as an query string argument I pass in the jsessionid for whichever domain the request is made to. For example if the domains have sessions as follows http://child.first-domain.com : JSESSIONID=1234 http://doctor.second-domain.com : JSESSIONID=ABCD http://nurse.third-domain.co.uk : JSESSIONID=wxyz a.jsp on child.first-domain.com html . a href=http://tracker.first-domain.com/track.gif?sid=1234 . /html a.jsp on doctor.second-domain.com html . a href=http://tracker.first-domain.com/track.gif?sid=ABCD . /html etc for nurse.third-domain.co.uk the servlet handling the tracker.first-domain.com/track.gif registers the different domain specific session ids with its own session id and thus ties them altogether, thus when one of the session ids logs on, its associated with a tracker session id which can then propogate the information to the other user sessions. Its a bit rough and ready, but seems to work, I think the tracker gif thing ist the most common way that the web analytics tools work. If you want more info on this I'll happily send you the code its a couple of java files about 400 lines in total. Thanks for your reply, All the best Simon - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: jk load balancing based upon ip address rather than session id
Ben, Rainer, That is an excellent idea, and would seem to be a very elegant solution, I'll give it a shot. Thank you both very much. I really appreciate it. Cheers Simon - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: jk load balancing based upon ip address rather than session id
Hi André, You could do the authentication and SSO handling at the Apache level, and set some partial domain cookie at that level, with some cross-domain identifier (as long as the domains have a common part of course). The browser will later send this cookie back with each request addressed to any of the servers that match the partial domain. Your understanding of it, as far as my understanding is concerned, is spot on, but like you say in your post ... as long as the domains have a common part of course Unfortunately in my case that doesn't hold true, its an international site, and we've got the same domain names for different tlds e.g.: mydomain.com mydomain.com.ar So right at the top level (com vs / .com.ar), I'm a bit hamstrung other than that you're completely correct. I read a few other posts about trying to ensure that the JSESSIONID cookie sent by TC applies to the domain name and not the subdomain through TC (rather than doing it through apache) - this one might be of interest to you: http://www.nabble.com/Share-session-cookie-across-subdomains-td16787390.html but regardless in my case this doesn't help me because I'm essentially using two distinct domains. Cheers for the comment though Simon - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: jk load balancing based upon ip address rather than session id
Hi Chris, Wow, does that really work? That's a tremendously cool hack, if so! It seems to, although it's not yet been tested in anger and there is scope for dependancy issues if you navigate from one domain directly to the another expecting a continous session experience. What I mean is that you need to have one request for a resource on each domain to ensure that that session id is tied to the tracking id. Although I can think of several simple ways to overcome this, although in my case I don't need to due to the structure business domain / use cases that the web app models. TC should not be respecting the JSESSIONID passed-in if it is invalid. Perhaps I'm misunderstanding you, though. Do you have a registry of session ids cross-linked in the session of each web application? If I understand you correctly, yes. I use the session ids as a convenient session unique ids in my own registry of user login sessions and a tracking id which happens to be a JSESSIONID. Cheers Simon - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
jk load balancing based upon ip address rather than session id
Hello, I'm using jk 1.2.25 with tomcat 5.5.25 and apache 2.0 on one debian box - 2.4.27-2-386 i686 GNU/Linux I've set up 3 tomcat instances that receive requests from the jk load balancer worker I've implemented in the web application, a simple cross domain single sign on (SSO) mechanism. This mechanism ties the different session ids in any single container together, regardless if they've originated from different domains, for example: sub1.mydomain.com sub2.yourdomain.com sub3.hisdomain.co.uk Hence when a user logs on this is then reflected in all the different sessions that they might be in a particular container for that user, from the serviced domains that they've visited. This is fine when there is just one container. But when there are several all servicing requests in a load balanced context, it doesn't work, because the session ids from different domains may be directed to different tomcat instances / containers, which then breaks the assumption that the SSO mechanism relies upon (that all sessions being held in a single container). The tomcat instances aren't in a distributed cluster and I'd like to keep it that way. My initial idea is to balance the traffic based upon the ip address rather than the session id, thus I can be assured that all requests from a particular ip address will hit the same container, and hence the single sign on mechanism will work. I realise that this would give me a much less granular balancing profile. I'd much prefer to do this through the jk load balancer although as an alternative I could do it through the balancer web app that comes with tomcat and implement a Rule in java myself. I'm open to other ideas that get the job done. I've read the following: http://tomcat.apache.org/connectors-doc/generic_howto/loadbalancers.html http://tomcat.apache.org/connectors-doc/reference/workers.html http://tomcat.apache.org/tomcat-5.5-doc/balancer-howto.html http://venus.rainbow-it.net/manual/en/mod/mod_proxy.html I've searched the forums as well, I had a cheeky look at the source too for shits and giggles, but still no joy. native/common/jk_lb_worker.c seems like the get_most_suitable_worker function is very much driven from the session id - I'm no a C programmer :-( My question is has anyone already done this? Is there a better way of doing it that I'm missing, or should I basically just do it through the balancer web in tomcat app and forego the jk load balancer, or bite the bullet and make the tomcat containers a cluster. Any suggestions / help would be very much appreciated Cheers Simon - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: jk load balancing based upon ip address rather than session id
Hi Chris, Thanks for the reply, Simon, Simon Papillon wrote: | when there are | several all servicing requests in a load balanced context, it doesn't | work, because the session ids from different domains may be directed | to different tomcat instances / containers, which then breaks the | assumption that the SSO mechanism relies upon (that all sessions being | held in a single container). | | The tomcat instances aren't in a distributed cluster and I'd like to | keep it that way. Isn't this what sticky sessions are for? You get randomly assigned to a server for your first request, and each subsequent request goes to that same server (unless it goes down, in which case you have to switch). This does not require distributable sessions. Does that not solve your SSO requirement? - -chris Forgive me if I'm overlooking something, but as far as I understand it, the sticky session mechanism is driven off the JSESSIONID that is assigned by the tomcat container when a client first makes a request that instigates a session creation, if no JSESSIONID cookie was sent as part of the request the node is chosen according to the worker.loadbalancer.method (Request, Session, Traffic, Busyness i think Request is the default) . Once a JSESSIONID has been set by a container the load balancer will then attach the JVMRoute onto the end which will then be used by the jk load balancer in future requests to determine the node to use to service the request. e.g. if I have three nodes (tomcatA, tomcatB, tomcatC) I could have the following scenario... child.first-domain.com : JSESSIONID = D75AA77AC6FBF43F2C2DDC195DDA6D44.tomcatC doctor.second-domain.com : JSESSIONID = 5D211C177DFB064DEF731832CF07D693.tomcatA nurse.third-domain.co.uk : JSESSIONID = E1EC672CAAA3F2F8348C2A23991DF46B.tomcatB Where my browser has made three seperate requests for three seperate resources, all serviced by the same group of tomcat containers through vhosting, behind the load balancer, in which case my SSO mechanism won't work as future requests on child.first-domain.com, doctor.second-domain.com and nurse.third-domain.co.uk will behandled by tomcatC, tomcatA and tomcatB respectively. As the SSO mechanism is based on the assumption that all requests from the same browser are handled by the same container, this will break the SSO, Let me know if I'm misunderstanding some fundermental way in which the jk load balancer works, or if I'm not explaing myself well, Cheers Simon - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
