Hi Filip, 3 Solaris machines, a variety of specs but one 7, one 8 and one 9 I think. As the machines are live we cant just chop and change the config but I am working on getting seperate tomcats set up on each of the 3 members in the original configuration to try and replicate the problems using Jmeter (which we havent managed yet).
"This is of course bad, increasing the "mcastDropTime" should prevent this." - I thought that but was a bit wary as 3 seconds seems like quite a long time as it is. It is of course an option and I might be able to replicate the problems easier by shortening this. Some timing info from one of the servers (Tomcat A) INFO: Average cluster serialize/send time=2780 ms for 5900 requests (16406080ms). INFO: Nr of bytes sent=68663955 over 20200 ==3399 bytes/request >From tomat B (The adding and dissappearing seems to happen a lot more with this one): INFO: Average cluster serialize/send time=356 ms for 84500 requests (30093367ms). INFO: Nr of bytes sent=2439724 over 1200 ==2033 bytes/request >From tomcat C: INFO: Average cluster serialize/send time=577 ms for 86200 requests (49785777ms). INFO: Nr of bytes sent=99673325 over 34900 ==2855 bytes/request On tomcat B there seems a lot of this: Jun 2, 2004 5:25:22 PM org.apache.catalina.cluster.tcp.SocketSender waitForAck WARNING: Wasn't able to read acknowledgement from server[/172.16.1.11:4001] in 15000 ms. Disconnecting socket, and trying again. and WARNING: Unable to send replicated message, is server down? java.net.SocketTimeoutException: Read timed out As a note i've been told that our sessions are larger than average. 50K or bigger has been suggested. We have only got about 70 users across the 3 nodes but the number of sessions seems to reach double this sometimes, perhaps because of user impatience and the cursed refresh button. If you know of a way to more accurately measure the session size then it might make this more concrete. We are also using the DeltaManager for session replication. Our load balancer is pen (which you recommended) and it works in a round robin fashion with sticky IPs. I have downloaded ThreadAnalyser and will get this installed tomorrow along with the new tomcat instances. Is there anything else that you'd like me to do in the meantime as well. It will benefit us greatly to get this resolved so I am glad to put some time in on it as well as hopefully benefitting all tomcat users. Ta Matt -----Original Message----- From: Filip Hanik - Dev [mailto:[EMAIL PROTECTED] Sent: 08 June 2004 16:54 To: Tomcat Users List Subject: Re: persist sessions in a clustered environment >members of the cluster rise so high that the machine became unusable. what OS are you on, I would be very interested to see what threads went high up. For Solaris there is a free tool called ThreadAnalyser which tells you exactly what the threads are doing. I am running load tests with hundreds of clients and I don't experience this CPU rise. >One thing I saw in the logs, which i'm unsure is the cause or a symptom, is that all >member >periodically saw other members of the cluster leaving then rejoining again 1 second later This is of course bad, increasing the "mcastDropTime" should prevent this. >My intended course of action was to retain the clustering >and write the sessions to a shared disk to Using the persistence manager and a shared file system, will achieve this. I of course, am more interested in figuring out what problems you are having and why :) since I haven't been able to reproduce those myself. Let me know if you would want to volunteer some of your time to work with me on this. I would also be interested in what your average cluster send times are and what the average message size is, should also have been printed to your logs. Filip ----- Original Message ----- From: "Dale, Matt" <[EMAIL PROTECTED]> To: "Tomcat Users List (E-mail)" <[EMAIL PROTECTED]> Sent: Tuesday, June 08, 2004 10:44 AM Subject: RE: persist sessions in a clustered environment Its not that the memory replication didnt work, it worked very well just seemed to have exteme problems with it under load, there may have been other factors though. Using clustering and in-memory replication we on fairly regular occasion saw the load of one or more members of the cluster rise so high that the machine became unusable. Turning off the clustering completly but keeping the load balancing seems to have sorted this. One thing I saw in the logs, which i'm unsure is the cause or a symptom, is that all member periodically saw other members of the cluster leaving then rejoining again 1 second later. I'm not sure what would cause this but I suppose too much network traffic from the in-memory session replication could cause a hold up, although I find it hard to believe that it would be over 3 seconds. Or as a cause of the problems, every time a member rejoins, all of the active sessions will have to be shared between the members so increased network traffic there. If you have any ideas on that i'd be grateful. My intended course of action was to retain the clustering and write the sessions to a shared disk to avoid the network traffic associated. My 1 question would be here, is that say I have 2 tomcats TomcatA and TomcatB, and they write their sessions to a shared file system using PersistenceManager, will each of them know about the sessions belonging to the other tomcat? Ta Matt > persist sessions in a clustered environmentuse the PersistenceManager instead of the > clustering if all you want is to write sessions to a file system. > http://jakarta.apache.org/tomcat/tomcat-5.0-doc/config/manager.html > Did you need any help figuring out why in memory replication didn't work? > Filip ----- Original Message ----- From: Dale, Matt To: Tomcat Users List (E-mail) Sent: Monday, June 07, 2004 4:03 PM Subject: persist sessions in a clustered environment Hi, We have been experiencing problems with our cluster using in-memory replication so we are looking into other ways in which to replicate the sessions. Our first step will be to switch to asynchronous mode for the replication from pooled but what i'm stuck with is replicating sessions to a shared file system. Basically I want to know how to do it or if someone can point me towards examples. Ta Matt ------------------------------------------------------------------------------ --------------------------------------------------------------------- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] -------------------------------------------------------------------------------- > --------------------------------------------------------------------- > To unsubscribe, e-mail: [EMAIL PROTECTED] > For additional commands, e-mail: [EMAIL PROTECTED] --------------------------------------------------------------------- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Any opinions expressed in this E-mail may be those of the individual and not necessarily the company. This E-mail and any files transmitted with it are confidential and solely for the use of the intended recipient. If you are not the intended recipient or the person responsible for delivering to the intended recipient, be advised that you have received this E-mail in error and that any use or copying is strictly prohibited. If you have received this E-mail in error please notify the beCogent postmaster at [EMAIL PROTECTED] Unless expressly stated, opinions in this email are those of the individual sender and not beCogent Ltd. You must take full responsibility for virus checking this email and any attachments. Please note that the content of this email or any of its attachments may contain data that falls within the scope of the Data Protection Acts and that you must ensure that any handling or processing of such data by you is fully compliant with the terms and provisions of the Data Protection Act 1984 and 1998.
--------------------------------------------------------------------- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
