RE: With JK 1.2.x using route causes problems with redirect?
Rainer (and all interested), Just thought I'd let you know I threw some load at 4 Tomcats behind 1 Apache with the JK connector using domains and routes to implement two different clusters of two Tomcats each and it seems to work well. My load test tool (Grinder) reported that I was executing about 23 transactions per second overall(however accurate that is). Not exhaustive testing, but it looks good so far ;-) Brian Horblit ---Original Message- --From: Horblit, Brian (TH USA) --Sent: Tuesday, September 18, 2007 12:30 PM --To: users@tomcat.apache.org --Subject: RE: With JK 1.2.x using route causes problems with redirect? -- -- --When I get multicast working in my Linux environment I'll --put it under load and do the same failover tests to see how --it fares when it is working harder! Maybe I'll do the same --for the route/redirect method too. -- -Original Message- From: Rainer Jung [mailto:[EMAIL PROTECTED] Sent: Tuesday, September 18, 2007 11:59 AM To: Tomcat Users List Subject: Re: With JK 1.2.x using route causes problems --with redirect? [EMAIL PROTECTED] wrote: Henk and Rainer, My sincerest apologies! I don't know how many times I --checked that config without noticing the error. Sorry to waste your time. Probably too many edits/tests wore my eyes out. :-( No problem. I learned something too (redirect=route), and found another tester for the domain feature :) I fixed that domain up and yes indeed, it seems to be working. I did 6 combinations of server fails and restarts with the expected behavior each time. Thanks once again, Brian Have fun! Rainer -- - To start a new topic, e-mail: users@tomcat.apache.org To --unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] -- --- --To start a new topic, e-mail: users@tomcat.apache.org To --unsubscribe, e-mail: [EMAIL PROTECTED] --For additional commands, e-mail: [EMAIL PROTECTED] -- -- - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: How do you access all sessions from a servlet in tomcat 6.0?
Gary, I'm not sure how much time was spent discussing alternatives as apposed to the question of if getting access to all sessions would work. You originally talked about tracking for concurrent license checking... We currently do this type of thing - and more, like abstracting the session from the http session so the http session can time out and return resources without actually losing the user session, concurrent connections versus concurrent users, etc. Furthermore we have multiple servers running the same app so that simply holding the information locally in the VM is not sufficient. We handle tracking all of this stuff via the database. Thus the distributed nature of our app and the precise lifecycle of the HttpSession are not so important. The downside to all this is that you have to be careful how you track activity via the db. How/when do you log activity? How do you write your connection-checking queries so you don't have performance issues? How/when do you cache stuff locally to help improve performance? How do you efficiently write logs for usage tracking? Using the DB works but it does take some tuning (as we found out ;-). Brian ---Original Message- --From: [EMAIL PROTECTED] --[mailto:[EMAIL PROTECTED] --Sent: Wednesday, September 19, 2007 5:05 AM --To: Tomcat Users List --Subject: Re: How do you access all sessions from a servlet --in tomcat 6.0? -- -- --Thanks for all the feedback on this. -- --___ -- --Gary Johnstone --Engineering IT --Cummins Turbo Technologies Ltd --+44 1484 440532 -- -- -- -- -- -- -- -- -- Lyallex -- -- [EMAIL PROTECTED] -- -- m -- To -- Tomcat Users List -- -- 19/09/2007 10:38 --users@tomcat.apache.org -- -- cc -- -- -- Please respond to -- Subject -- Tomcat Users Re: How do you access --all sessions -- List from a servlet in --tomcat 6.0? -- [EMAIL PROTECTED] -- -- che.org -- -- -- -- -- -- -- -- -- -- -- -- -- --OK, for some reason I've been obsessing about this for a --whole day now. -- --If you hold an external reference to a Session then --according to my tests the session will still time out as --expected but the external reference will be non null. At the --very least this means that you may end up with a large --number of useless references taking up space in memory. Of --course you can always remove an invalid or timed out --reference in the sessionDestroyed method of your listener. -- --There are a whole bunch of other issues surrounding this but --I'm sure you've sussed them out for yourself already. -- --Anyway, I'll shut up now. -- --Rgds --Duncan -- -- --On 9/19/07, Lyallex [EMAIL PROTECTED] wrote: -- On 9/18/07, Lyallex [EMAIL PROTECTED] wrote: -- How about creating a SessionListener -- -- class SomeSessionListener implements HttpSessionListener ... -- -- Register it in web.xml -- -- in the sessionCreated method of your listener get a --reference to the -- new session from the HttpSessionEvent you can now access the -- getLastAccessedTime(), maybe store the refs in some singleton ... -- -- ...er, actually I think this could be a REALLY STUPID idea --as I hadn't -- thought about what happens if you maintain an external --reference to a -- session and the session expires... -- -- Investigating now -- -- Duncan -- -- --- --To start a new topic, e-mail: users@tomcat.apache.org To --unsubscribe, e-mail: [EMAIL PROTECTED] --For additional commands, e-mail: [EMAIL PROTECTED] -- -- -- --_ --This e-mail transmission and any attachments to it are --intended solely for the use of the individual or entity to --whom it is addressed and may contain confidential and --privileged information. If you are not the intended --recipient, your use, forwarding, printing, storing, --disseminating, distribution, or copying of this --communication is prohibited. If you received this --communication
RE: With JK 1.2.x using route causes problems with redirect?
Henk and Rainer, My sincerest apologies! I don't know how many times I checked that config without noticing the error. Sorry to waste your time. Probably too many edits/tests wore my eyes out. :-( I fixed that domain up and yes indeed, it seems to be working. I did 6 combinations of server fails and restarts with the expected behavior each time. Thanks once again, Brian ---Original Message- --From: Rainer Jung [mailto:[EMAIL PROTECTED] --Sent: Tuesday, September 18, 2007 10:13 AM --To: Tomcat Users List --Subject: Re: With JK 1.2.x using route causes problems with redirect? -- --Henk Fictorie wrote: -- -- Brian.Horblit wrote: -- -- # Set w4 properties -- worker.w4.socket_keepalive=1 -- worker.w4.socket_timeout=20 -- worker.w4.reply_timeout=2 -- worker.w4.retries=2 -- worker.w4.connection_pool_timeout=60 -- worker.w4.type=ajp13 -- worker.w4.host=localhost -- worker.w4.port=8034 -- worker.w4.lbfactor=1 -- worker.w4.route=tc4 -- worker.w3.domain=d34 -- -- -- -- I guess this is a typo, it should be worker.w4.domain=d34 -- -- regards Henk -- --And I already wondered what the heck was wrong with domains. --Thanks! So --Brian: give it another try with the fixed typo. I don't see --anything else wrong in this configuration. -- --Regards -- --Rainer -- --- --To start a new topic, e-mail: users@tomcat.apache.org To --unsubscribe, e-mail: [EMAIL PROTECTED] --For additional commands, e-mail: [EMAIL PROTECTED] -- -- - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: With JK 1.2.x using route causes problems with redirect?
When I get multicast working in my Linux environment I'll put it under load and do the same failover tests to see how it fares when it is working harder! Maybe I'll do the same for the route/redirect method too. ---Original Message- --From: Rainer Jung [mailto:[EMAIL PROTECTED] --Sent: Tuesday, September 18, 2007 11:59 AM --To: Tomcat Users List --Subject: Re: With JK 1.2.x using route causes problems with redirect? -- --[EMAIL PROTECTED] wrote: -- Henk and Rainer, -- -- My sincerest apologies! I don't know how many times I checked that -- config without noticing the error. Sorry to waste your --time. Probably -- too many edits/tests wore my eyes out. :-( -- --No problem. I learned something too (redirect=route), and --found another tester for the domain feature :) -- -- I fixed that domain up and yes indeed, it seems to be --working. I did 6 -- combinations of server fails and restarts with the --expected behavior -- each time. -- -- Thanks once again, -- -- Brian -- --Have fun! -- --Rainer -- --- --To start a new topic, e-mail: users@tomcat.apache.org To --unsubscribe, e-mail: [EMAIL PROTECTED] --For additional commands, e-mail: [EMAIL PROTECTED] -- -- - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: With JK 1.2.x using route causes problems with redirect?
Rainer, I thought I should just note that part of the exotic config (using route and the like) is driven by the fact that the workers.properties file is not dynamically reloadable. We have had a modified version of the JK2 connector running for years and we heavily use the capability to dynamically reload the workers2.properties file to update worker activation status during the deploy of new app versions. I was mulling over how to get similar functionality (without apachectl/graceful) via multiple load balancers pointing to workers in different states and switching things out dynamically at the url=lb mapping level. (I realize that I can change states via the jkstatus page, but I want the changes to persistent.) I think I saw in the bug database discussion of making the workers.properties dynamically reloadable, and I wanted to cast a vote for that feature. Thanks again for everything, Brian Horblit ---Original Message- --From: Rainer Jung [mailto:[EMAIL PROTECTED] --Sent: Tuesday, September 18, 2007 11:59 AM --To: Tomcat Users List --Subject: Re: With JK 1.2.x using route causes problems with redirect? -- --[EMAIL PROTECTED] wrote: -- Henk and Rainer, -- -- My sincerest apologies! I don't know how many times I checked that -- config without noticing the error. Sorry to waste your --time. Probably -- too many edits/tests wore my eyes out. :-( -- --No problem. I learned something too (redirect=route), and --found another tester for the domain feature :) -- -- I fixed that domain up and yes indeed, it seems to be --working. I did 6 -- combinations of server fails and restarts with the --expected behavior -- each time. -- -- Thanks once again, -- -- Brian -- --Have fun! -- --Rainer -- --- --To start a new topic, e-mail: users@tomcat.apache.org To --unsubscribe, e-mail: [EMAIL PROTECTED] --For additional commands, e-mail: [EMAIL PROTECTED] -- -- - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
With JK 1.2.x using route causes problems with redirect?
All, Good day! For a little background, I'm working on implementing sub-clusters with the new connector. That is, I will cluster my Tomcats in paris to keep network traffic/memory usage down. The connector would preferrentially fail-over between sub-cluster Tomcat pairs. But if both elements of a Tomcat cluster go down, I'd like the connector to fail-over to another sub-cluster (with session loss). I realize that the connector Domain directive was meant for this. My reading of the docs, however, would imply that since the jvmRoute is the domain (with sticky sessions), the requests going into a domain would be load-balanced between domain memembers. This would in turn imply (for those of us who are paranoid), that we'd have to do synchronous session replication at the Tomcat level to *ensure* that under heavy load sessions would be replicated before a user is balanced between cluster members. This would then, potentially, slow responses to the client since sending the response would be delayed until session replication is complete. (Perhaps I should try everything to confirm all this since the docs are a wee bit vagues...) So, I thought I'd be sneaky and try other directives. Hence: I am trying to use JK 1.2.25 to access Tomcat 6. I am setting up 4 servers paired in two different clusters at the Tomcat level. In the workers.properties I have: - 4 workers: w1,w2,w3,w4 - All 4 workers are sub-workers of a load balancer called router - The 4 workers are configured as two pairs where the worker pairs preferrentially fail-over to each other using the redirect directive. - In order to prepare for some more exotic configuration, I have used the route directive to de-couple the worker name from the route/jvmRoute which I use for sticky load balancing. - The exotic config will be to define multiple load balancers so I can switch them out at runtime via the url=worker mapping file (since one can't reload the workers.properties file at runtime, which is a bummer!). I have found that the redirect directive is not working all the time. If w1 and w2 are set to preferentially fail-over to each other, it does not always happen. For example, if I kill the Tomcat w1 is pointing to, the connector will at times fail-over to the Tomcat w4 is pointing to. If I remove the route directive from all workers and ensure the worker names match the jvmRoute in Tomcat, all is well - I don't get the incorrect fail-overs. It seems then the redirect pairs always failover to each other. Note that the problem seems to appear after I do a failover that behaves properly once, restart the failed servers, and try another failover test. Is my config incorrect? Do the route and redirect directives not play well with each other? I've tried this on Windoze (see config below) with all Tomcats on my desktop. I've also tried in on Linux with the Tomcats on 4 different servers. Same behaviour. Thanks very much in advance! My workers.properties is as follows: ### Global worker maintenance interval in seconds worker.maintain=30 ### ### The list of all workers ### worker.list=router ### ### The real workers ### # Set w1 properties worker.w1.socket_keepalive=1 worker.w1.socket_timeout=20 worker.w1.reply_timeout=2 worker.w1.retries=2 worker.w1.connection_pool_timeout=60 worker.w1.type=ajp13 worker.w1.host=localhost worker.w1.port=8031 worker.w1.lbfactor=1 worker.w1.redirect=w2 worker.w1.route=tc1 # Set w2 properties worker.w2.socket_keepalive=1 worker.w2.socket_timeout=20 worker.w2.reply_timeout=2 worker.w2.retries=2 worker.w2.connection_pool_timeout=60 worker.w2.type=ajp13 worker.w2.host=localhost worker.w2.port=8032 worker.w2.lbfactor=1 worker.w2.redirect=w1 worker.w2.route=tc2 # Set w3 properties worker.w3.socket_keepalive=1 worker.w3.socket_timeout=20 worker.w3.reply_timeout=2 worker.w3.retries=2 worker.w3.connection_pool_timeout=60 worker.w3.type=ajp13 worker.w3.host=localhost worker.w3.port=8033 worker.w3.lbfactor=1 worker.w3.redirect=w4 worker.w3.route=tc3 # Set w4 properties worker.w4.socket_keepalive=1 worker.w4.socket_timeout=20 worker.w4.reply_timeout=2 worker.w4.retries=2 worker.w4.connection_pool_timeout=60 worker.w4.type=ajp13 worker.w4.host=localhost worker.w4.port=8034 worker.w4.lbfactor=1 worker.w4.redirect=w3 worker.w4.route=tc4 # router is a load balancer worker.router.type=lb worker.router.balance_workers=w1,w2,w3,w4 worker.router.sticky_session=True worker.router.sticky_session_force=False worker.router.method=S worker.router.recover_time=30 ### ### The status worker ### worker.list=jkstatus worker.jkstatus.type=status Brian D. Horblit Senior Principal Engineer Thomson Healthcare (303) 486-6697 (800) 525-9083 x 6697 www.thomsonhealthcare.com http://www.thomsonhealthcare.com/ [EMAIL PROTECTED]
RE: With JK 1.2.x using route causes problems with redirect?
Rainer, Thanks very much for the quick response! You guys are awesome. So, as I now understand it, the domain mechanism is compatable with (and independent of) the route. And even if domains are used, the stickyness of sticky sessions is at the server level since the route is used to find individual servers. I'll give this a try right away! Out of curiosity, is the issue I noted (incorrect redirct failover when route used) a known issue? Even though it is not intended usage should I submit a big report? Thanks very much again - I appreciate the quick and detailed response! Brian Horblit ---Original Message- --From: Rainer Jung [mailto:[EMAIL PROTECTED] --Sent: Monday, September 17, 2007 9:56 AM --To: Tomcat Users List --Subject: Re: With JK 1.2.x using route causes problems with redirect? -- --hi Brian, -- --[EMAIL PROTECTED] wrote: -- All, -- -- Good day! -- -- For a little background, I'm working on implementing --sub-clusters -- with the new connector. That is, I will cluster my Tomcats --in paris to -- keep network traffic/memory usage down. The connector would -- preferrentially fail-over between sub-cluster Tomcat pairs. But if -- both elements of a Tomcat cluster go down, I'd like the --connector to -- fail-over to another sub-cluster (with session loss). -- -- I realize that the connector Domain directive was meant --for this. My -- reading of the docs, however, would imply that since the --jvmRoute is -- the domain (with sticky sessions), the requests going into --a domain -- would be load-balanced between domain memembers. -- --jvmRoute - worker.xxx.route --domain influences failover decision, as you want it. -- -- This would in turn imply (for -- those of us who are paranoid), that we'd have to do synchronous -- session replication at the Tomcat level to *ensure* that --under heavy -- load sessions would be replicated before a user is --balanced between -- cluster members. This would then, potentially, slow --responses to the -- client since sending the response would be delayed until session -- replication is complete. (Perhaps I should try everything --to confirm -- all this since the docs are a wee bit vagues...) So, I --thought I'd be -- sneaky and try other directives. Hence: -- --You can set jvmRoute to the worker name (or as you suggest --the worker route), use session stickyness and relate --differrent workers by using the same domain name. -- --The if all workers are up, stickyness will be effective, if --a worker dies, JK will choose another worker in the same --domain, and only if the whole domain is dead choose any other worker. -- --I'm pretty sure, that you can simulate this somehow with --redirect, but the domain concept also works, once you --increase the domain size. At the end both concepts are very similar. -- -- I am trying to use JK 1.2.25 to access Tomcat 6. I am setting up 4 -- servers paired in two different clusters at the Tomcat --level. In the -- workers.properties I have: -- -- - 4 workers: w1,w2,w3,w4 -- - All 4 workers are sub-workers of a load balancer called router -- - The 4 workers are configured as two pairs where the --worker pairs -- preferrentially fail-over to each other using the --redirect directive. -- - In order to prepare for some more exotic configuration, --I have used -- the route directive to de-couple the worker name from the -- route/jvmRoute which I use for sticky load balancing. -- - The exotic config will be to define multiple load --balancers so I can -- switch them out at runtime via the url=worker mapping --file (since one -- can't reload the workers.properties file at runtime, which is a -- bummer!). -- --You can do apachectl graceful. Switching via --uriworkermap.properties should be fine though. -- -- I have found that the redirect directive is not working --all the time. -- If w1 and w2 are set to preferentially fail-over to each other, it -- does not always happen. For example, if I kill the Tomcat w1 is -- pointing to, the connector will at times fail-over to the --Tomcat w4 is pointing to. -- -- If I remove the route directive from all workers and ensure the -- worker names match the jvmRoute in Tomcat, all is well - I --don't get -- the incorrect fail-overs. It seems then the redirect --pairs always -- failover to each other. -- -- Note that the problem seems to appear after I do a failover that -- behaves properly once, restart the failed servers, and try another -- failover test. -- -- Is my config incorrect? Do the route and redirect --directives not -- play well with each other? -- -- I've tried this on Windoze (see config below) with all --Tomcats on my -- desktop. I've also tried in on Linux with the Tomcats on 4 --different -- servers. Same behaviour. -- -- Thanks very much in advance! -- -- My workers.properties is as follows: -- -- -- ### Global worker maintenance interval in seconds --worker.maintain=30 -- -- ### --
RE: With JK 1.2.x using route causes problems with redirect?
Rainer, How about a big bug report? Sorry for the typo ;-) An update. With the route/redirect strategy, I did as you suggested and changed my redirect to the route name rather than the worker name. It now seems to work as I itended: Failovers are between redirect partners. I did a good 6 or 7 failures, back and forth, with no errors. Thanks very much! I then went is and simply deleted the redirect directive and put in a domain directive. Everything else stayed the same. That is I had workers w1-w4, each of which had a specified route tc1-tc4. This did not work. It basically behaved like my redirect configuration when I specified worker names rather than routes for the redirect. At times fail-overs went outside the domain. My config for this scenario is below. If I understood your other email, all I needed to change was swapping domain for redirect and I could leave the routes alone, so that is what I tried. Thanks again! === workers.proprties for attempt at using both domain and route ### Global worker maintenance interval in seconds worker.maintain=30 ### ### The list of all workers ### worker.list=router ### ### The real workers ### # Set w1 properties worker.w1.socket_keepalive=1 worker.w1.socket_timeout=20 worker.w1.reply_timeout=2 worker.w1.retries=2 worker.w1.connection_pool_timeout=60 worker.w1.type=ajp13 worker.w1.host=localhost worker.w1.port=8031 worker.w1.lbfactor=1 worker.w1.route=tc1 worker.w1.domain=d12 # Set w2 properties worker.w2.socket_keepalive=1 worker.w2.socket_timeout=20 worker.w2.reply_timeout=2 worker.w2.retries=2 worker.w2.connection_pool_timeout=60 worker.w2.type=ajp13 worker.w2.host=localhost worker.w2.port=8032 worker.w2.lbfactor=1 worker.w2.route=tc2 worker.w2.domain=d12 # Set w3 properties worker.w3.socket_keepalive=1 worker.w3.socket_timeout=20 worker.w3.reply_timeout=2 worker.w3.retries=2 worker.w3.connection_pool_timeout=60 worker.w3.type=ajp13 worker.w3.host=localhost worker.w3.port=8033 worker.w3.lbfactor=1 worker.w3.route=tc3 worker.w3.domain=d34 # Set w4 properties worker.w4.socket_keepalive=1 worker.w4.socket_timeout=20 worker.w4.reply_timeout=2 worker.w4.retries=2 worker.w4.connection_pool_timeout=60 worker.w4.type=ajp13 worker.w4.host=localhost worker.w4.port=8034 worker.w4.lbfactor=1 worker.w4.route=tc4 worker.w3.domain=d34 # router is a load balancer worker.router.type=lb worker.router.balance_workers=w1,w2,w3,w4 worker.router.sticky_session=True worker.router.sticky_session_force=False worker.router.method=S worker.router.recover_time=30 ### ### The status worker ### worker.list=jkstatus worker.jkstatus.type=status ---Original Message- --From: Rainer Jung [mailto:[EMAIL PROTECTED] --Sent: Monday, September 17, 2007 11:40 AM --To: Tomcat Users List --Subject: Re: With JK 1.2.x using route causes problems with redirect? -- --Hi Brian, -- --[EMAIL PROTECTED] schrieb: -- Out of curiosity, is the issue I noted (incorrect --redirct failover -- when route used) a known issue? Even though it is not --intended usage -- should I submit a big report? -- --I think we neither need a bug report, nor a big report :) -- --Quick shot: redirect needs to be a route, not a worker name. --Since you decided to map routes to workers via the route --attribute, you most likely need to use the route name --instead of the worker name in the redirect attribute. At --least that's how the code looks like. -- --Redirect is older than route, and we didn't think about the --implication, when adding route. It's not bad enough to --justify a bug report, I think. --We should note it in the docs though. -- --Let us know, if the resulting configuration works for you --(whatever you prefer, redirect or domain). -- --Regards, -- --Rainer -- --- --To start a new topic, e-mail: users@tomcat.apache.org To --unsubscribe, e-mail: [EMAIL PROTECTED] --For additional commands, e-mail: [EMAIL PROTECTED] -- -- - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: JK Loadbalancer not balancing fairly
Ben, So I assume you have two web servers fronting two app servers - or there are two servers both of which have a web server and an app server? For the restart you talk about - did you restart both web servers? Do you have a good load balancer (local director, content director like an F5) in front of the two web servers? If I am reading your JKStatus text correctly I noticed the following: Load balancer value on web server 2 --- = ~0.56 Load balancer value on web server 1 but Number requests on web server 2 --- = ~0.91 Number requests on web server 1 Now, if I am interpreting the meaning of load balancer value and number of reuqests correctly, that would imply that the number of sessions stuck to each app server from web server 1 is very roughly twice as high as from 2, but the total number of requests sent to each app server from both web servers is very roughly the same. (Can someone confirm I'm intrepreting those #s correctly?) According to the docs, each connect by default trys to keep the number of requests sent to each worker the same, which looks to be happening reasonably well. (I'm playing with trying the keep the number of sessions balanced since our apps tend to be more of a memory issue than a cpu issue. There is a setting on the connector for this.) With a some info on your setup we can try to figure out the load imbalance. As a note, I am playing with the jk1.2.x connector, but our productio systems use the old jk2.x connector. With that, I've seen a load imbalance on the app servers when one of the app serves has gone down for a while, and then has come back up. If the connectors are not reset, they will try to catch up the restarted app server in terms of the number of requests it has handled, thus loading it more heavily than servers that have been up the whole time. Brian ---Original Message- --From: ben short [mailto:[EMAIL PROTECTED] --Sent: Thursday, August 23, 2007 4:51 AM --To: Tomcat Users List --Subject: JK Loadbalancer not balancing fairly -- --Hi All, -- --We are doing some load testing on our setup and find that --the cpu use age of tomcat reported by top on the two systems --is not equal. --Typically we see figures like ~400% to 800% cpu on one --machine and ~50% on the other machine for the java process. --We would expect that the two cpu values to be equal. -- --The jkstatus page on box one shows the following after a restart. --Although before a restart the Max column was showing 250 for --jcpres1 and 32 for jcpres2. -- --Name TypeHostAddrAct State D F --MV Acc Err CE RE Wr Rd --Busy Max Route RR Cd Rs -- jcpres1 ajp13 172.16.4.11:8009 --172.16.4.11:8009 ACT OK 0 1 1 --869 42762 4 0 939K286M1 --11 jcpres1 -- 0/0 -- jcpres2 ajp13 172.16.4.12:8009 --172.16.4.12:8009 ACT OK 0 1 1 --869 42772 1 0 943K280M2 --9jcpres2 -- 0/0 -- --and box 2 -- --Name TypeHostAddrAct State D F --MV Acc Err CE RE Wr Rd --Busy Max Route RR Cd Rs -- jcpres1 ajp13 172.16.4.11:8009 --172.16.4.11:8009 ACT OK 0 1 1 --484 38720 4 0 850K256M3 --10 jcpres1 -- 0/0 -- jcpres2 ajp13 172.16.4.12:8009 --172.16.4.12:8009 ACT OK 0 1 1 --483 38710 4 0 850K260M1 --10 jcpres2 -- 0/0 -- -- --Our system setup. -- --Both machines are running the the following software on RedHat 4ES -- --Httpd 2.2.4 --Mod JK 1.2.25 --Tomcat 6.0.12 --Java 1.6.0_01 -- --Box 1. -- --workers.properties -- --# JK Status worker config -- --worker.list=jkstatus --worker.jkstatus.type=status -- --# Presentaton Load Balancer Config -- --worker.list=preslb -- --worker.preslb.type=lb --worker.preslb.balance_workers=jcpres1,jcpres2 --worker.preslb.sticky_session=1 -- --worker.jcpres1.port=8009 --worker.jcpres1.host=172.16.4.11 --worker.jcpres1.type=ajp13 --worker.jcpres1.lbfactor=1 --worker.jcpres1.fail_on_status=503,400,500,909 -- --worker.jcpres2.port=8009 --worker.jcpres2.host=172.16.4.12 --worker.jcpres2.type=ajp13 --worker.jcpres2.lbfactor=1 --worker.jcpres2.fail_on_status=503,400,500,909 -- -- --Box 2. -- --workers.properties -- --# JK Status worker config -- --worker.list=jkstatus --worker.jkstatus.type=status -- --# Presentaton Load Balancer Config -- --worker.list=preslb -- --worker.preslb.type=lb --worker.preslb.balance_workers=jcpres1,jcpres2 --worker.preslb.sticky_session=1 -- --worker.jcpres1.port=8009 --worker.jcpres1.host=172.16.4.11 --worker.jcpres1.type=ajp13
RE: JK Loadbalancer not balancing fairly
Rainer, Thanks very much for the clarification! Since I have playing with the load balancing strategy set to session (worker.router.method=S on my load balancer), is there a way to tell roughly how many sessions have been pinned to each worker/tomcat? In this case would the load balancer value be (something like) the number of new sessions sent to a particular worker divided by two some number of times? If this were true you still would not know the number of sessions pinned to a worked because of the factors of two having been divided out. I just got a HTTP JMX adapter wired up in Tomcat so I'll see if I can get session info that way... Thanks again, Brian ---Original Message- --From: Rainer Jung [mailto:[EMAIL PROTECTED] --Sent: Thursday, August 23, 2007 11:22 AM --To: Tomcat Users List --Subject: Re: JK Loadbalancer not balancing fairly -- --[EMAIL PROTECTED] schrieb: -- Ben, -- -- So I assume you have two web servers fronting two app servers - or -- there are two servers both of which have a web server and an app -- server? For the restart you talk about - did you restart both web -- servers? Do you have a good load balancer (local director, content -- director like an F5) in front of the two web servers? -- -- If I am reading your JKStatus text correctly I noticed the --following: -- -- Load balancer value on web server 2 -- --- = ~0.56 Load balancer --value on web -- server 1 -- -- but -- -- Number requests on web server 2 -- --- = ~0.91 Number requests on web -- server 1 -- -- -- Now, if I am interpreting the meaning of load balancer value and -- number of reuqests correctly, that would imply that the --number of -- sessions stuck to each app server from web server 1 is --very roughly -- twice as high as from 2, but the total number of requests --sent to each -- app server from both web servers is very roughly the same. (Can -- someone confirm I'm intrepreting those #s correctly?) -- --The number of requests is the total since last jk/apache --restart. So if the last restart was shortly before, the --numbers will not help. If they were not reset after the --tests, we would know, that Apache 1 had a little more --requests than apache 2, but both of them send exacty the --same number of requests to the two tomcat nodes (delta=1 request). -- --The V column is the balancing value used to decide, where --the next request goes to. It is the number of requests sent --to the tomcat divided by two once a minute, so it is --multiplied by a decay curve. The big difference between the --V values of apache 1 and apache 2 does not matter. It could --simply mean, that the one with the bigger V value did it's --division more recent in time. The V values for the two --tomcats are again very similar on the same Apache, another --indication of good balancing. -- --All his is true for the default balancing method Requests. -- --I would suggest first to follow CPU by Tomcat process over --the test period (not per system and not simply as one --number, instead as a graph over time). -- -- According to the docs, each connect by default trys to --keep the number -- of requests sent to each worker the same, which looks to --be happening -- reasonably well. (I'm playing with trying the keep the number of -- sessions balanced since our apps tend to be more of a memory issue -- than a cpu issue. There is a setting on the connector for this.) -- -- With a some info on your setup we can try to figure out the load -- imbalance. -- -- As a note, I am playing with the jk1.2.x connector, but --our productio -- systems use the old jk2.x connector. With that, I've seen a load -- imbalance on the app servers when one of the app serves --has gone down -- for a while, and then has come back up. If the connectors are not -- reset, they will try to catch up the restarted app --server in terms -- of the number of requests it has handled, thus loading it --more heavily -- than servers that have been up the whole time. -- --The catchup problem should be fixed. A recovered or --reactivated worker gets the biggest work done value of all --other workers, so it should start normal or even a little --less loaded. -- -- -- Brian -- --Regards, -- --Rainer -- --- --To start a new topic, e-mail: users@tomcat.apache.org To --unsubscribe, e-mail: [EMAIL PROTECTED] --For additional commands, e-mail: [EMAIL PROTECTED] -- -- - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: Does anyone have an approach to checking if Tomcat instance is UP?
Kim, You mentiond fool-proof... Perhaps a multi-pronged approach is best, if you have the time and inclination to implement it. 1) Apps can have issues for lots of reasons (running out of memory, db load and/or locks, thread deadlocks, etc, etc.) In lots of cases the VM/Tomcat are OK, but the app is not. With this in mind, if you can get your monitoring software to actually hit the app's core functionality itself, you can see if the app is responding, no matter if the VM/Tomcat are OK. Can you get your monitoring software to hit a non-trivial, non-access-controlled page? Can you get the monitor to login/logout every so often (say every few minutes)? How about building a health-check page in the app itself that queries all major resources like RDBMSs, and full text search engines, and LDAPs, etc and prints out status for all those? Depending on our access control setup, you can restrict access to the status page to your monitor (perpahs via IP address) or simply have that URL innaccessable from the public network. Also consider your tolerence for slowness. If the site is up but taking 20 seconds to respond, maybe that is broken. So of the monitor can check for response times that will help. 2) For the OOME condition, Jeff beat me to it. You can also, for example, use JMX and an http adapter to query an Mbean for the current memory state and alert if you are above 90%. The alert can be an email. This pro-active approach (and a fast response to the alert ;-) means your app won't get to the point of being unresponsive. 3) Perhaps this is too obvious, but why is the app running out of memory? Is there a leak or is the number of simultaneous requests too much? Have you given the app enough headroom in terms of max heap? We've had apps that just suck memory like nuts and have many simultaneous users, but if the VM is sized large enough they are happy. 4) For (free) monitoring software that can help with some of this, check out http://www.nagios.org/. Nagios and Splunk are cool. It is awesome what this stuff does for the price! Hope this helps at least a little, Brian Horblit Brian D. Horblit Senior Principal Engineer Thomson Healthcare (303) 486-6697 (800) 525-9083 x 6697 www.thomsonhealthcare.com [EMAIL PROTECTED] ---Original Message- --From: Jeff Hoffmann [mailto:[EMAIL PROTECTED] --Sent: Monday, August 20, 2007 10:25 AM --To: Tomcat Users List --Subject: Re: Does anyone have an approach to checking if --Tomcat instance is UP? [stuff deleted...] --If what you're concerned about is an OOME, you can have a --JSP that queries the runtime memory usage and outputs --something easily parsable to alert you to a (pending) problem. IE: -- --Runtime rt = Runtime.getRuntime(); --double used = rt.totalMemory()-rt.freeMemory(); double free --= rt.freeMemory(); double available = --rt.maxMemory()-rt.totalMemory(); double usedpercent = --(used/rt.maxMemory()) * 100; double freepercent = --(free/rt.maxMemory()) * 100; double availablepercent = --(available/rt.maxMemory()) * 100; -- --If your normal state is used % is 50%, you could send out --a warning when it goes to 70% or restart if it goes to 90%. -- --Jeff Hoffmann --Head Plate Spinner --PropertyKey.com -- --- --To start a new topic, e-mail: users@tomcat.apache.org To --unsubscribe, e-mail: [EMAIL PROTECTED] --For additional commands, e-mail: [EMAIL PROTECTED] -- -- - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Apache Tomcat Connector connection pool's relation to workers and processes?
First, thanks in advance! I have been playing with the 1.2.23 version of the JK Connector. In particular, I've been setting up an Apache 2 front end to multiple Tomcat 6's by using load balancing workers delegating to real workers. For the most part the documentation is quite clear, but something I've not found explictly discussed (perhaps I've just missed it???) is the relation between the connector's connection pool and the workers. Discussions of the connection pool note that a pool is set up for each Apache child process and the number of connections in the pool should correspond to the number of threads in the child process (for threaded Apaches). Each child process will have its own instance of the connector and the connector's load balancer worker and real workers (I would assume). - Are the connections in the connection pool simply dolled out as needed to real workers, across all workers and their target Tomcats? - In the (old new) JK2 connector, there was a setting to limit the number of connections from a worker to its target (the Tomcat instance). Looks like there is no such option on JK 1.2.x? If not, one would be able to limit the overall number of connections (from and Apache process) to all the workers/Tomcats via the connection_pool_size, and one would be able to throttle the number of simultaneous requests Tomcat would attempt to handle via the maxProcessors attribute. I liked the ability to throttle the traffic to the worker (on the web server) so I just wanted to ensure I understood the JK model. - Should the number of connections really match the Apache child process threads? The Apache threads might need to be set high to handle static content requests and traffic spikes that you don't want Tomcat to deal with... Thanks again! Brian Brian D. Horblit Senior Principal Engineer Thomson Healthcare (303) 486-6697 (800) 525-9083 x 6697 www.thomsonhealthcare.com http://www.thomsonhealthcare.com/ [EMAIL PROTECTED]
