Re: Tuning session replication on clusters
On 5 Sep 2012, at 20:51, llow...@oreillyauto.com llow...@oreillyauto.com wrote: I have a small cluster of 3 nodes running tomcat 6.0.24 with openJDK 1.6.0_20 on Ubuntu 10.04 LTS. You should really consider upgrading Java. While you're at it, try Tomcat 7.0.30 in your lab and have a look at BackupManager (instead of DeltaManager) as Dan suggested. p I have roughly 5,000-6,000 sessions at any given time, and when I restart one of the nodes I am finding that not all sessions are getting replicated , even when I have the state transfer timeout set to 60 seconds. It seems that only sessions that have been touched recently are replicated, even if the session is still otherwise valid. I did one test where I created about 1,500 sessions and then took out one node, When I brought it back online, it only replicated the 4-5 sessions that were from active users on the test cluster. It did not replicated the idle sessions that were still valid that my prior test had created. I am wanting to tune my settings, but I am unsure where would be the best place to start. Should I start with the threads available to the NIO Receiver, or would I be better off focusing on a different set of attributes first, such as the send or receive timeout values? Any tips or pointers as to which setting might be the most productive would be greatly appreciated. Lee Lowder O'Reilly Auto Parts Web Systems Administrator (417) 862-2674 x1858 This communication and any attachments are confidential, protected by Communications Privacy Act 18 USCS � 2510, solely for the use of the intended recipient, and may contain legally privileged material. If you are not the intended recipient, please return or destroy it immediately. Thank you. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tuning session replication on clusters
Hi Kyle, Great testing, btw. So when you say x5, did you change the settings as follows: rxBufAize=125940 (= 25188 x 5) By any chance, have you analyzed a heapdump of Tomcat at periodic intervals to see which class is hogging heap during the session replication? Thanks. -Shanti On Fri, Sep 7, 2012 at 12:19 PM, kharp...@oreillyauto.com wrote: Chris: Assembling the sessions into a Collection is likely to be very fast, since it's just copying references around: the size of the individual sessions should not matter. Of course, pushing all those bytes to the other servers... Perhaps Tomcat does something like serialize the session to a big binary structure and then sends that (which sounds insane -- streaming binary data is how that should be done -- but I haven't checked to code to be sure). It appears that tomcat is serializing all the data into a singular structure, rather than a collection of references. Watching VisualVM plot heap usage during replication, it nearly doubles (in my test env, this was the only thing running so that makes sense). If you're sure Tomcat is only making references, then I'd propose there is a problem with the JVM dereferencing the collection elements and double-counting the memory used. Either way, it's enough to make the JVM report a doubling of heap usage and a raise to the heap allocation. As soon as replication is done, heap use goes back to normal. I've attached a screenshot to the zip file. Now for data: I did tests of 200 sessions (~20 MB) at a time (200, 400, 600... up to 3000). I then tested in groups of 1000 (3000, 4000, 5000... up to 10k). At no point did I receive any exceptions or OOME issues. Heap usage never climbed above 60% Xmx. My lab was isolated to help give consistent results. Here are some points. 1. There is a pivotal point where replication performance degrades dramatically. In my tests, this happened around 2400-2600 sessions. I restarted tomcat and was able to avoid the issue, until I hit 2800 sessions (~300 MB total session data). There was a 153% jump in time required to perform replication at this point. From there, each subsequent test took marginally longer per session (15-25%) than the test before it. Chris was correct, it's not exponential, but the ms/session gets worse and worse as we climb. I have no explanation for the sharp jump or the continued degradation as we climb. I've seem similar performance issues with sort and comparative logic, but those don't make sense here. Perhaps this serialized object is being jerked around Young Gen/Old Gen and having to be constantly reallocated? Grasping at straws here... 2. Networking is a large portion of the bottleneck for large data sets. The thread size and pool size attributes to the sender/receiver had no impact on throughput. Also, a packet capture revealed nothing naughty happening. However, the rxBufSize and txBufSize values on the Nio receiver and the PooledParallel transport elements made a profound difference. I generated 7000 sessions (~700MB) and used default settings: 74 sec. Increasing the rx/tx settings by x5 I was able to replicate the sessions in 33 sec. Gains beyond x5 were almost nil; at x100 (which is absurd) only resulted in 29.3 sec replication. A simple SCP transfer of a 700 MB file (using tmpfs folders) between these same two systems took 13 seconds. My conclusion is that tuning the network was obviously a great help, but it still took 30 seconds to replicate 700MB worth of session data on a network with enough throughput to perform the transfer in 13 seconds. I don't know if further network settings could be changed for the DeltaManager to aid in speeding up replication, but given the spike in memory use and the pivotal performance drop at a consistent point I'm inclined to think we're hitting some edge case regarding session size and memory settings (Xmx/Heap and NewSize/SurvivorRatio). As Chris said, if Tomcat isn't collecting just references, it probably should be. Feel free to pick apart my data or thoughts. I tried to be as analytical as possible, but there's a lot of conjecture in here. Attachment (See attached file: SessionResearch.zip) If the list strips it, find copy here: https://docs.google.com/open?id=0B876X8DOwh8peEkyZVd6RVc4cWc Thanks. Kyle Harper This communication and any attachments are confidential, protected by Communications Privacy Act 18 USCS § 2510, solely for the use of the intended recipient, and may contain legally privileged material. If you are not the intended recipient, please return or destroy it immediately. Thank you. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tuning session replication on clusters
Shanti: So when you say x5, did you change the settings as follows: rxBufAize=125940 (= 25188 x 5) Yes, x5 means default values (25188 and 43800) times 5. Each value was extrapolated in the spreadsheet so you can see them there too. By any chance, have you analyzed a heapdump of Tomcat at periodic intervals to see which class is hogging heap during the session replication? No, but if it will help I can try to do this. I'm hoping to get a good amount of feedback and group any more testing that needs done together into a single night. Kyle Harper This communication and any attachments are confidential, protected by Communications Privacy Act 18 USCS § 2510, solely for the use of the intended recipient, and may contain legally privileged material. If you are not the intended recipient, please return or destroy it immediately. Thank you. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tuning session replication on clusters
Shanti Suresh sha...@umich.edu wrote on 09/07/2012 12:37:34 PM: From: Shanti Suresh sha...@umich.edu To: Tomcat Users List users@tomcat.apache.org Date: 09/07/2012 12:44 PM Subject: Re: Tuning session replication on clusters Hi Kyle, Great testing, btw. So when you say x5, did you change the settings as follows: rxBufAize=125940 (= 25188 x 5) I work with Kyle on this, and was the OP for this thread. When he says x5, what he did was set the rx/tx values to 5 times their original values, so yes, if the original value was 25188 then the x5 value would be 125940. We figured that using multiples of the original values would make for the simplest analysis and for more accurate comparisons. By any chance, have you analyzed a heapdump of Tomcat at periodic intervals to see which class is hogging heap during the session replication? Thanks. -Shanti On Fri, Sep 7, 2012 at 12:19 PM, kharp...@oreillyauto.com wrote: Chris: Assembling the sessions into a Collection is likely to be very fast, since it's just copying references around: the size of the individual sessions should not matter. Of course, pushing all those bytes to the other servers... Perhaps Tomcat does something like serialize the session to a big binary structure and then sends that (which sounds insane -- streaming binary data is how that should be done -- but I haven't checked to code to be sure). It appears that tomcat is serializing all the data into a singular structure, rather than a collection of references. Watching VisualVM plot heap usage during replication, it nearly doubles (in my test env, this was the only thing running so that makes sense). If you're sure Tomcat is only making references, then I'd propose there is a problem with the JVM dereferencing the collection elements and double-counting the memory used. Either way, it's enough to make the JVM report a doubling of heap usage and a raise to the heap allocation. As soon as replication is done, heap use goes back to normal. I've attached a screenshot to the zip file. Now for data: I did tests of 200 sessions (~20 MB) at a time (200, 400, 600... up to 3000). I then tested in groups of 1000 (3000, 4000, 5000... up to 10k). At no point did I receive any exceptions or OOME issues. Heap usage never climbed above 60% Xmx. My lab was isolated to help give consistent results. Here are some points. 1. There is a pivotal point where replication performance degrades dramatically. In my tests, this happened around 2400-2600 sessions. I restarted tomcat and was able to avoid the issue, until I hit 2800 sessions (~300 MB total session data). There was a 153% jump in time required to perform replication at this point. From there, each subsequent test took marginally longer per session (15-25%) than the test before it. Chris was correct, it's not exponential, but the ms/session gets worse and worse as we climb. I have no explanation for the sharp jump or the continued degradation as we climb. I've seem similar performance issues with sort and comparative logic, but those don't make sense here. Perhaps this serialized object is being jerked around Young Gen/Old Gen and having to be constantly reallocated? Grasping at straws here... 2. Networking is a large portion of the bottleneck for large data sets. The thread size and pool size attributes to the sender/receiver had no impact on throughput. Also, a packet capture revealed nothing naughty happening. However, the rxBufSize and txBufSize values on the Nio receiver and the PooledParallel transport elements made a profound difference. I generated 7000 sessions (~700MB) and used default settings: 74 sec. Increasing the rx/tx settings by x5 I was able to replicate the sessions in 33 sec. Gains beyond x5 were almost nil; at x100 (which is absurd) only resulted in 29.3 sec replication. A simple SCP transfer of a 700 MB file (using tmpfs folders) between these same two systems took 13 seconds. My conclusion is that tuning the network was obviously a great help, but it still took 30 seconds to replicate 700MB worth of session data on a network with enough throughput to perform the transfer in 13 seconds. I don't know if further network settings could be changed for the DeltaManager to aid in speeding up replication, but given the spike in memory use and the pivotal performance drop at a consistent point I'm inclined to think we're hitting some edge case regarding session size and memory settings (Xmx/Heap and NewSize/SurvivorRatio). As Chris said, if Tomcat isn't collecting just references, it probably should be. Feel free to pick apart my data or thoughts. I tried to be as analytical as possible, but there's a lot of conjecture in here. Attachment (See attached file: SessionResearch.zip) If the list strips it, find copy here: https
Re: Tuning session replication on clusters
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Kyle, On 9/7/12 12:19 PM, kharp...@oreillyauto.com wrote: Chris: Assembling the sessions into a Collection is likely to be very fast, since it's just copying references around: the size of the individual sessions should not matter. Of course, pushing all those bytes to the other servers... Perhaps Tomcat does something like serialize the session to a big binary structure and then sends that (which sounds insane -- streaming binary data is how that should be done -- but I haven't checked to code to be sure). It appears that tomcat is serializing all the data into a singular structure, rather than a collection of references. :( Watching VisualVM plot heap usage during replication, it nearly doubles (in my test env, this was the only thing running so that makes sense). That certainly sound like Tomcat is chewing-through a lot of heap space. Without understanding the implementation, I can't comment on whether or not that is really necessary, but I have to imagine that a streaming-serialization (or at least buffered, where one session is serialized at a time and then streamed) would be superior. If you're sure Tomcat is only making references, then I'd propose there is a problem with the JVM dereferencing the collection elements and double-counting the memory used. It's very unlikely that the JVM is making that kind of mistake. Also, I haven't looked at a single line of Tomcat's session-distribution code so I'm not in a position to make accurate assertions about its implementation. Either way, it's enough to make the JVM report a doubling of heap usage and a raise to the heap allocation. As soon as replication is done, heap use goes back to normal. I've attached a screenshot to the zip file. Sounds like your analysis is reasonable. I'll look at your data and make further comments. In the meantime, I've heard over the years one particular thing from people I feel know what they are talking about: don't use HttpSession. It's not that the servlet spec's session-management isn't useful, it's that it is very hard to make it scale properly when using most container-managed clustering implementations. First, it almost always uses Java Serialization which is not terribly efficient. Second, it is very coarse-grained: you replicate the entire session or the whole thing fails (unless you use some vendor-specific tricks like Rainer's suggestion of suppressing certain attributes). If you have a lot of data in the session that doesn't *need* to be replicated (like caches, etc.) then you might be wasting a lot of time. So, what to use instead? Memcached comes up often as a similar solution to the problem of clustered temporary data storage. Why memcached instead of HttpSession clustering? I think about half of the answer is that when you have to manually push your own data to a remote cache (instead of allowing the container to handle the details), you tend to get very frugal with the amount of data you end up pushing. We don't use distributed sessions on our product even though (almost) everything in the session is serializable. Even so, I've thought of writing a wrapper for certain objects (like caches) that we store in the session that never need to be replicated (at least not in their complete forms). I haven't done it because, well, we don't need it. Stick-sessions with failover is good enough for us, and we don't have to saturate our network with session-replication chatter. - -chris -BEGIN PGP SIGNATURE- Version: GnuPG/MacGPG2 v2.0.17 (Darwin) Comment: GPGTools - http://gpgtools.org Comment: Using GnuPG with Mozilla - http://www.enigmail.net/ iEYEARECAAYFAlBKN9oACgkQ9CaO5/Lv0PB3LACgsrVWsuWWkb0ckfIPeiNUMoq4 8fcAoIb0FQU/2EsET1AmIHGkX20si4lG =xKjd -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tuning session replication on clusters
Try jmap , jps,they will tell you about the usage of memory if you have doubt and monitor network data transfered 在 2012-9-6，13:59，kharp...@oreillyauto.com 写道： Alright, I did some more testing with another application and found the following: SessTime (sec 100.101 1250.101 5000.201 15000.201 18000.101 24000.101 42,0000.901 (that's not a typo) Turns out the application that was having trouble is storing a silly amount of crap in the session. I am still not 100% sure what's happening behind the scenes at the 1500 session mark, but I'm guessing that based on our session size (nearly 700 MB) and memory configuration we were hitting some heap ceiling and the replication was forced to 'juggle'. If anyone has any more background on what's happening feel free to set me straight. I'll check back later... I need to go beat some developers... Kyle Harper From:kharp...@oreillyauto.com To:Tomcat Users List users@tomcat.apache.org Date:09/05/2012 07:55 PM Subject:Re: Tuning session replication on clusters I'm working with Lee on this as well, so I can help answer most of that. In short: Yes, all our replication is working well. We have keepalived acting as a vrrp device (no round-robin dns) in front of a few web servers (apache 2.2.x, mod_proxy/mod_ajp) which are using stickysessions and BalancerMembers. Replication (DeltaManager/SimpleTCPCluster) is working as intended on the tomcat side (6.0.24). After further research, the problem we're seeing is performance with replication when the number of sessions is larger than around 2000. Using Jmeter on our test servers I can reproduce the problem. Here are the times it takes to replicate X number of sessions when an application is restarted: Sess Time (sec) 10 0.101 125 0.401 500 1.302 1500 2.104 1800 5.308 1800 6.709 2400 15.02 3600 30.285 3600 27.238 The times make sense until around 1500. The time it takes to replicate more than 1500 sessions becomes exponentially worse. Here is our cluster configuration from node1: Engine name=Catalina defaultHost=localhost jvmRoute=tntest-app-a-1 Cluster className=org.apache.catalina.ha.tcp.SimpleTcpCluster channelSendOptions=8 Manager className=org.apache.catalina.ha.session.DeltaManager stateTransferTimeout=45 expireSessionsOnShutdown=false notifyListenersOnReplication=true / Channel className=org.apache.catalina.tribes.group.GroupChannel Membership className=org.apache.catalina.tribes.membership.McastService address=239.255.0.1 port=45564 frequency=500 dropTime=3000 / Receiver className=org.apache.catalina.tribes.transport.nio.NioReceiver address=auto port=4000 autoBind=100 selectorTimeout=5000 maxThreads=6 / Sender className=org.apache.catalina.tribes.transport.ReplicationTransmitter Transport className=org.apache.catalina.tribes.transport.nio.PooledParallelSender timeout=45000 / /Sender Interceptor className=org.apache.catalina.tribes.group.interceptors.TcpFailureDetector/ Interceptor className=org.apache.catalina.tribes.group.interceptors.MessageDispatch15Interceptor/ /Channel Valve className=org.apache.catalina.ha.tcp.ReplicationValve filter=/ Valve className=org.apache.catalina.ha.session.JvmRouteBinderValve/ ClusterListener className=org.apache.catalina.ha.session.JvmRouteSessionIDBinderListener/ ClusterListener className=org.apache.catalina.ha.session.ClusterSessionListener/ /Cluster The best time we got for 3600 sessions was 24 seconds, and that's when I added the following to the Manager tag (stole this from the 5.5 docs; not even sure it's valid in 6.x): sendAllSessions=false sendAllSessionsSize=500 sendAllSessionsWait=20 What has me stumped is why the time required to do more sessions is exponentially higher beyond 1500 sessions. Using JMeter I can simulate 3600 new users (all creating a session) and the two servers can serve the requests AND generate/replicate the sessions in under 19 seconds. Any ideas would be greatly appreciated. I have a full test environment to simulate anything you might recommend. Sincerely, Kyle Harper From: Igor Cicimov icici...@gmail.com To: Tomcat Users List users@tomcat.apache.org Date: 09/05/2012 07:12 PM Subject: Re: Tuning session replication on clusters On Thu, Sep 6, 2012 at 5:51 AM, llow...@oreillyauto.com wrote: I have a small cluster of 3
Re: Tuning session replication on clusters
On Sep 5, 2012, at 8:48 PM, kharp...@oreillyauto.com wrote: I'm working with Lee on this as well, so I can help answer most of that. In short: Yes, all our replication is working well. We have keepalived acting as a vrrp device (no round-robin dns) in front of a few web servers (apache 2.2.x, mod_proxy/mod_ajp) which are using stickysessions and BalancerMembers. Replication (DeltaManager/SimpleTCPCluster) is working as intended on the tomcat side (6.0.24). Jumping in a little late on this thread, but have you considered trying the BackupManager instead of DeltaManager? The DeltaManager is going to replicate session data to all cluster members while BackupManager will only replicate to one backup cluster member. This might save you some time on restart. https://tomcat.apache.org/tomcat-7.0-doc/config/cluster-manager.html#org.apache.catalina.ha.session.BackupManager_Attributes Dan After further research, the problem we're seeing is performance with replication when the number of sessions is larger than around 2000. Using Jmeter on our test servers I can reproduce the problem. Here are the times it takes to replicate X number of sessions when an application is restarted: Sess Time (sec) 100.101 125 0.401 500 1.302 1500 2.104 1800 5.308 1800 6.709 2400 15.02 3600 30.285 3600 27.238 The times make sense until around 1500. The time it takes to replicate more than 1500 sessions becomes exponentially worse. Here is our cluster configuration from node1: Engine name=Catalina defaultHost=localhost jvmRoute=tntest-app-a-1 Cluster className=org.apache.catalina.ha.tcp.SimpleTcpCluster channelSendOptions=8 Manager className=org.apache.catalina.ha.session.DeltaManager stateTransferTimeout=45 expireSessionsOnShutdown=false notifyListenersOnReplication=true / Channel className=org.apache.catalina.tribes.group.GroupChannel Membership className=org.apache.catalina.tribes.membership.McastService address=239.255.0.1 port=45564 frequency=500 dropTime=3000 / Receiver className=org.apache.catalina.tribes.transport.nio.NioReceiver address=auto port=4000 autoBind=100 selectorTimeout=5000 maxThreads=6 / Sender className=org.apache.catalina.tribes.transport.ReplicationTransmitter Transport className=org.apache.catalina.tribes.transport.nio.PooledParallelSender timeout=45000 / /Sender Interceptor className=org.apache.catalina.tribes.group.interceptors.TcpFailureDetector/ Interceptor className=org.apache.catalina.tribes.group.interceptors.MessageDispatch15Interceptor/ /Channel Valve className=org.apache.catalina.ha.tcp.ReplicationValve filter=/ Valve className=org.apache.catalina.ha.session.JvmRouteBinderValve/ ClusterListener className=org.apache.catalina.ha.session.JvmRouteSessionIDBinderListener/ ClusterListener className=org.apache.catalina.ha.session.ClusterSessionListener/ /Cluster The best time we got for 3600 sessions was 24 seconds, and that's when I added the following to the Manager tag (stole this from the 5.5 docs; not even sure it's valid in 6.x): sendAllSessions=false sendAllSessionsSize=500 sendAllSessionsWait=20 What has me stumped is why the time required to do more sessions is exponentially higher beyond 1500 sessions. Using JMeter I can simulate 3600 new users (all creating a session) and the two servers can serve the requests AND generate/replicate the sessions in under 19 seconds. Any ideas would be greatly appreciated. I have a full test environment to simulate anything you might recommend. Sincerely, Kyle Harper From: Igor Cicimov icici...@gmail.com To: Tomcat Users List users@tomcat.apache.org Date: 09/05/2012 07:12 PM Subject: Re: Tuning session replication on clusters On Thu, Sep 6, 2012 at 5:51 AM, llow...@oreillyauto.com wrote: I have a small cluster of 3 nodes running tomcat 6.0.24 with openJDK 1.6.0_20 on Ubuntu 10.04 LTS. I have roughly 5,000-6,000 sessions at any given time, and when I restart one of the nodes I am finding that not all sessions are getting replicated , even when I have the state transfer timeout set to 60 seconds. It seems that only sessions that have been touched recently are replicated, even if the session is still otherwise valid. I did one test where I created about 1,500 sessions and then took out one node, When I brought it back online, it only replicated the 4-5 sessions that were from active users on the test cluster. It did not replicated the idle sessions that were still valid that my
Re: Tuning session replication on clusters
Thanks all for replies (and for the jmap/jps idea, hadn't thought of that for some reason). I tried increasing the maxThreads on the NioReceviever and noticed no performance gain. I then modified the poolSize on the Transport element to 100 and saw no performance gain. This actually didn't surprise me after I discovered how large the sessions were. Using JMX (VisualVM) I watched the Heap size on my two servers as I tested 7000 sessions. Heap climbed approximately 1GB. When I restarted node2, I watched node1's heap usage nearly double. This confirmed my suspicion that the replication process is putting a copy of all sessions into a new object (list I suppose?) before transmitting them. After replication finished (109 seconds), node1's heap usage went back to normal. The aggregation of sessions into a new object to be sent (I presume as part of the handleGET_ALL_SESSIONS?) seems to work quickly, though I'm not sure how to test how much of the 109 seconds it took to replicate was Tomcat gathering up all the sessions to send and how much was network traffic. We have a low utilization gigabit ethernet fabric connecting all servers, so transferring 1GB of data shouldn't take more than 10-12 seconds. Does anyone know if there are ways to time the different steps in the replication process? If it is the network send/receive process that's slow, are there transmit/receive settings for the sender/receiver that could aid in speeding up replication of large data chunks? I see there are rxBufSize and txBufSize settings on the Receiver and Transport elements, and they're set to 25/43kb. If those values represents how data is chunked then larger settings might help (similar to the throughput difference of transferring 100x 10MB files vs. 10,000x 100kb files on a network). Any feedback is always appreciated. I will be able to test any suggestions later tonight and I'll be sure to report back if we make some headway. I'm sure this topic would be useful to others in the future. Thanks, Kyle Harper What has me stumped is why the time required to do more sessions is exponentially higher beyond 1500 sessions. Using JMeter I can simulate 3600 new users (all creating a session) and the two servers can serve the requests AND generate/replicate the sessions in under 19 seconds. Any ideas would be greatly appreciated. I have a full test environment to simulate anything you might recommend. Maybe that's the boundary for the 6 threads used for the messages between the cluster members, having in mind the huge size of your sessions? By default the Sender uses 25 simultanious connections to each of the other nodes so maybe increasing this pool might speed up the things (poolSize value inside the Transport element of the Sender)? This communication and any attachments are confidential, protected by Communications Privacy Act 18 USCS § 2510, solely for the use of the intended recipient, and may contain legally privileged material. If you are not the intended recipient, please return or destroy it immediately. Thank you. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tuning session replication on clusters
On 06.09.2012 15:10, kharp...@oreillyauto.com wrote: ... This actually didn't surprise me after I discovered how large the sessions were. Using JMX (VisualVM) I watched the Heap size on my two servers as I tested 7000 sessions. Heap climbed approximately 1GB. When I restarted node2, I watched node1's heap usage nearly double. This confirmed my suspicion that the replication process is putting a copy of all sessions into a new object (list I suppose?) before transmitting them. After replication finished (109 seconds), node1's heap usage went back to normal. That's a plausible explanation for your observation. You can split replication in several chunks using the config items you already observed. Even in TC 6 the DeltaManager supports: sendAllSessions (Default: true, means all session send in one message, false means split in multiple messages) sendAllSessionsSize (Default: 1000, number of sessions send per message when switch is false) sendAllSessionsWaitTime (Default: 2000; sleep pause between sending consecutive messages) The aggregation of sessions into a new object to be sent (I presume as part of the handleGET_ALL_SESSIONS?) seems to work quickly, though I'm not sure how to test how much of the 109 seconds it took to replicate was Tomcat gathering up all the sessions to send and how much was network traffic. We have a low utilization gigabit ethernet fabric connecting all servers, so transferring 1GB of data shouldn't take more than 10-12 seconds. Does anyone know if there are ways to time the different steps in the replication process? Set log level of org.apache.catalina.ha.session.DeltaManager to DEBUG or FINE depending whether you are using log4j or juli for Tomcat. If it is the network send/receive process that's slow, Try sniffing both ends for network analysis. are there transmit/receive settings for the sender/receiver that could aid in speeding up replication of large data chunks? I see there are rxBufSize and txBufSize settings on the Receiver and Transport elements, and they're set to 25/43kb. If those values represents how data is chunked then larger settings might help (similar to the throughput difference of transferring 100x 10MB files vs. 10,000x 100kb files on a network). Regards, Rainer - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tuning session replication on clusters
On 06.09.2012 16:57, Rainer Jung wrote: On 06.09.2012 15:10, kharp...@oreillyauto.com wrote: ... This actually didn't surprise me after I discovered how large the sessions were. Using JMX (VisualVM) I watched the Heap size on my two servers as I tested 7000 sessions. Heap climbed approximately 1GB. When I restarted node2, I watched node1's heap usage nearly double. This confirmed my suspicion that the replication process is putting a copy of all sessions into a new object (list I suppose?) before transmitting them. After replication finished (109 seconds), node1's heap usage went back to normal. That's a plausible explanation for your observation. You can split replication in several chunks using the config items you already observed. Even in TC 6 the DeltaManager supports: sendAllSessions (Default: true, means all session send in one message, false means split in multiple messages) sendAllSessionsSize (Default: 1000, number of sessions send per message when switch is false) sendAllSessionsWaitTime (Default: 2000; sleep pause between sending consecutive messages) I forgot one more thing: since TC 6.0.34 and 7.0.22 is it possible to decide which session attributes get replicated. So in case you have only few attributes that make up most of the big session memory *and* your application is able to transparently handle the situation, that these attributes are suddenly missing from the session, e.g. by retrieving the data again from some back end system or database, the following might help: Look for sessionAttributeFilter in http://tomcat.apache.org/tomcat-6.0-doc/config/cluster-manager.html I'm not saying it is easy, but if you want to make your application using session replication really efficient, it is a possible way to go. In addition there is a way an application can detect whether there was a node fail over, ie. a request is handled by another node as the previous request for the same session. You can hook filling missing attributes on this detection. The detection uses a feature of the ReplicationValve, which can set a request attribute that can be inspected to decide whether there was a fail over. Look for primaryIndicator in http://tomcat.apache.org/tomcat-6.0-doc/config/cluster-valve.html. If the attribute is false, you just switched nodes (fail over) and are now working on a replicated session. Regards, Rainer - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tuning session replication on clusters
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Kyle, On 9/5/12 9:59 PM, kharp...@oreillyauto.com wrote: Alright, I did some more testing with another application and found the following: SessTime (sec 10 0.101 125 0.101 500 0.201 1500 0.201 1800 0.101 24000.101 42,0000.901 (that's not a typo) Good to know that Tomcat itself does not seem to be the problem: thanks for the report. One question: After further research, the problem we're seeing is performance with replication when the number of sessions is larger than around 2000. Using Jmeter on our test servers I can reproduce the problem. Here are the times it takes to replicate X number of sessions when an application is restarted: Sess Time (sec) 10 0.101 125 0.401 500 1.302 1500 2.104 1800 5.308 1800 6.709 2400 15.02 3600 30.285 3600 27.238 The times make sense until around 1500. The time it takes to replicate more than 1500 sessions becomes exponentially worse. Plot those as X-Y in a spreadsheet and you'll see that it's only a bit worse than linear, especially after 1500. There's no enough data presented to draw an exponential performance curve conclusion: you're going to need more data to see if this is worse than linear. Can you collect more data? I know that creating 42k sessions of 700MiB each is probably a .. challenge, but more data would certainly be helpful. - -chris -BEGIN PGP SIGNATURE- Version: GnuPG/MacGPG2 v2.0.17 (Darwin) Comment: GPGTools - http://gpgtools.org Comment: Using GnuPG with Mozilla - http://www.enigmail.net/ iEYEARECAAYFAlBIwxkACgkQ9CaO5/Lv0PCrxwCeLMDD9Zx7PW6sdhnF+K+ONP79 HP0AoLfdb6NTm8xGHRimYqdqap1oQKTX =x6Z6 -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tuning session replication on clusters
Rainer: Thanks for the input. I'll do some additional testing with the sendAllSessions attributes, but my initial testing didn't show much gain. If the rx/tx settings are already chunking up the session bytes into smallish payloads then nothing I change with the sendAllSessions will improve that. I might be completely off base, but I'm unsure of what the rxBufSize and txBufSize attributes do exactly, and if changing them would boost throughput. Chris: One question: Plot those as X-Y in a spreadsheet and you'll see that it's only a bit worse than linear, especially after 1500. There's no enough data presented to draw an exponential performance curve conclusion: you're going to need more data to see if this is worse than linear. Can you collect more data? I know that creating 42k sessions of 700MiB each is probably a .. challenge, but more data would certainly be helpful. Before I change anything (tuning) I'll do as much testing as I can tonight to generate a data series and graph. Btw, does this list support .ods/.xls attachments? I'm not sure how Tomcat handles building the object containing all the sessions to send (Collection/List?), but it'll be interesting to see if there are points where the basic nature of the Collection object becomes less efficient. The testing so far has already shown that the quantity of sessions (like the 42k test I did in 901 ms on a different app) doesn't appear to be an issue, while large session sizes do. I know that the JVM was forced to make a full copy of all the session data for the application inside heap space too, so after certain points I might be reaching contention with memory management processes that handle heap size, ratios, garbage collection, etc. Having seen how quickly the JVM ramped up memory usage for replication, I'm relatively confident the session aggregation and duplication into a new Collection on the sender side is happening fast... realy fast. I'll try to get a protocol analyzer in place to capture the packets and compare them to a basic file transfer of a single large file. If anyone knows much about those rxBufSize and txBufSize settings and whether they'll impact anything let me know. If they operate like TCP congestion and window sizes then adjusting them could have profound performance implications. Thanks! Kyle Harper This communication and any attachments are confidential, protected by Communications Privacy Act 18 USCS § 2510, solely for the use of the intended recipient, and may contain legally privileged material. If you are not the intended recipient, please return or destroy it immediately. Thank you. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tuning session replication on clusters
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Kyle, On 9/6/12 2:45 PM, kharp...@oreillyauto.com wrote: Chris: One question: Plot those as X-Y in a spreadsheet and you'll see that it's only a bit worse than linear, especially after 1500. There's no enough data presented to draw an exponential performance curve conclusion: you're going to need more data to see if this is worse than linear. Can you collect more data? I know that creating 42k sessions of 700MiB each is probably a .. challenge, but more data would certainly be helpful. Before I change anything (tuning) I'll do as much testing as I can tonight to generate a data series and graph. Btw, does this list support .ods/.xls attachments? I'm not sure. You might be better off wrapping the file in a ZIP file (which is silly, since OpenDocument is already ZIP, but it might help) or just throwing the file up on pastebin (which I generally don't like because the message goes into the archives but pastebin items expire, etc. Just try not to drop a multi-MiB file onto the list ;) I'm not sure how Tomcat handles building the object containing all the sessions to send (Collection/List?), but it'll be interesting to see if there are points where the basic nature of the Collection object becomes less efficient. The testing so far has already shown that the quantity of sessions (like the 42k test I did in 901 ms on a different app) doesn't appear to be an issue, while large session sizes do. Assembling the sessions into a Collection is likely to be very fast, since it's just copying references around: the size of the individual sessions should not matter. Of course, pushing all those bytes to the other servers... I know that the JVM was forced to make a full copy of all the session data for the application inside heap space too, so after certain points I might be reaching contention with memory management processes that handle heap size, ratios, garbage collection, etc. Perhaps Tomcat does something like serialize the session to a big binary structure and then sends that (which sounds insane -- streaming binary data is how that should be done -- but I haven't checked to code to be sure). Having seen how quickly the JVM ramped up memory usage for replication, I'm relatively confident the session aggregation and duplication into a new Collection on the sender side is happening fast... realy fast. I'll try to get a protocol analyzer in place to capture the packets and compare them to a basic file transfer of a single large file. Sounds good. - -chris -BEGIN PGP SIGNATURE- Version: GnuPG/MacGPG2 v2.0.17 (Darwin) Comment: GPGTools - http://gpgtools.org Comment: Using GnuPG with Mozilla - http://www.enigmail.net/ iEYEARECAAYFAlBJI7YACgkQ9CaO5/Lv0PDcIwCfYaKt6ddWk3Nok6bxR2RX8ouX XEYAni+8N/oGPx/vl9GTwPfVXEek0u/Y =Tdtz -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Tuning session replication on clusters
I have a small cluster of 3 nodes running tomcat 6.0.24 with openJDK 1.6.0_20 on Ubuntu 10.04 LTS. I have roughly 5,000-6,000 sessions at any given time, and when I restart one of the nodes I am finding that not all sessions are getting replicated , even when I have the state transfer timeout set to 60 seconds. It seems that only sessions that have been touched recently are replicated, even if the session is still otherwise valid. I did one test where I created about 1,500 sessions and then took out one node, When I brought it back online, it only replicated the 4-5 sessions that were from active users on the test cluster. It did not replicated the idle sessions that were still valid that my prior test had created. I am wanting to tune my settings, but I am unsure where would be the best place to start. Should I start with the threads available to the NIO Receiver, or would I be better off focusing on a different set of attributes first, such as the send or receive timeout values? Any tips or pointers as to which setting might be the most productive would be greatly appreciated. Lee Lowder O'Reilly Auto Parts Web Systems Administrator (417) 862-2674 x1858 This communication and any attachments are confidential, protected by Communications Privacy Act 18 USCS § 2510, solely for the use of the intended recipient, and may contain legally privileged material. If you are not the intended recipient, please return or destroy it immediately. Thank you. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tuning session replication on clusters
On Thu, Sep 6, 2012 at 5:51 AM, llow...@oreillyauto.com wrote: I have a small cluster of 3 nodes running tomcat 6.0.24 with openJDK 1.6.0_20 on Ubuntu 10.04 LTS. I have roughly 5,000-6,000 sessions at any given time, and when I restart one of the nodes I am finding that not all sessions are getting replicated , even when I have the state transfer timeout set to 60 seconds. It seems that only sessions that have been touched recently are replicated, even if the session is still otherwise valid. I did one test where I created about 1,500 sessions and then took out one node, When I brought it back online, it only replicated the 4-5 sessions that were from active users on the test cluster. It did not replicated the idle sessions that were still valid that my prior test had created. I am wanting to tune my settings, but I am unsure where would be the best place to start. Should I start with the threads available to the NIO Receiver, or would I be better off focusing on a different set of attributes first, such as the send or receive timeout values? Any tips or pointers as to which setting might be the most productive would be greatly appreciated. Lee Lowder O'Reilly Auto Parts Web Systems Administrator (417) 862-2674 x1858 This communication and any attachments are confidential, protected by Communications Privacy Act 18 USCS § 2510, solely for the use of the intended recipient, and may contain legally privileged material. If you are not the intended recipient, please return or destroy it immediately. Thank you. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org For starter does your cluster satisfy the requirements bellow? To run session replication in your Tomcat 6.0 container, the following steps should be completed: - All your session attributes must implement java.io.Serializable - Uncomment the Cluster element in server.xml - If you have defined custom cluster valves, make sure you have the ReplicationValve defined as well under the Cluster element in server.xml - If your Tomcat instances are running on the same machine, make sure the tcpListenPort attribute is unique for each instance, in most cases Tomcat is smart enough to resolve this on it's own by autodetecting available ports in the range 4000-4100 - Make sure your web.xml has the distributable/ element - If you are using mod_jk, make sure that jvmRoute attribute is set at your Engine Engine name=Catalina jvmRoute=node01 and that the jvmRoute attribute value matches your worker name in workers.properties - Make sure that all nodes have the same time and sync with NTP service! - Make sure that your loadbalancer is configured for sticky session mode. Also you don't say what are you using for load balancing? Not bad to post your cluster definition as well.
Re: Tuning session replication on clusters
I'm working with Lee on this as well, so I can help answer most of that. In short: Yes, all our replication is working well. We have keepalived acting as a vrrp device (no round-robin dns) in front of a few web servers (apache 2.2.x, mod_proxy/mod_ajp) which are using stickysessions and BalancerMembers. Replication (DeltaManager/SimpleTCPCluster) is working as intended on the tomcat side (6.0.24). After further research, the problem we're seeing is performance with replication when the number of sessions is larger than around 2000. Using Jmeter on our test servers I can reproduce the problem. Here are the times it takes to replicate X number of sessions when an application is restarted: Sess Time (sec) 10 0.101 125 0.401 500 1.302 15002.104 18005.308 18006.709 240015.02 360030.285 360027.238 The times make sense until around 1500. The time it takes to replicate more than 1500 sessions becomes exponentially worse. Here is our cluster configuration from node1: Engine name=Catalina defaultHost=localhost jvmRoute=tntest-app-a-1 Cluster className=org.apache.catalina.ha.tcp.SimpleTcpCluster channelSendOptions=8 Manager className=org.apache.catalina.ha.session.DeltaManager stateTransferTimeout=45 expireSessionsOnShutdown=false notifyListenersOnReplication=true / Channel className=org.apache.catalina.tribes.group.GroupChannel Membership className=org.apache.catalina.tribes.membership.McastService address=239.255.0.1 port=45564 frequency=500 dropTime=3000 / Receiver className=org.apache.catalina.tribes.transport.nio.NioReceiver address=auto port=4000 autoBind=100 selectorTimeout=5000 maxThreads=6 / Sender className=org.apache.catalina.tribes.transport.ReplicationTransmitter Transport className=org.apache.catalina.tribes.transport.nio.PooledParallelSender timeout=45000 / /Sender Interceptor className=org.apache.catalina.tribes.group.interceptors.TcpFailureDetector/ Interceptor className=org.apache.catalina.tribes.group.interceptors.MessageDispatch15Interceptor/ /Channel Valve className=org.apache.catalina.ha.tcp.ReplicationValve filter=/ Valve className=org.apache.catalina.ha.session.JvmRouteBinderValve/ ClusterListener className=org.apache.catalina.ha.session.JvmRouteSessionIDBinderListener/ ClusterListener className=org.apache.catalina.ha.session.ClusterSessionListener/ /Cluster The best time we got for 3600 sessions was 24 seconds, and that's when I added the following to the Manager tag (stole this from the 5.5 docs; not even sure it's valid in 6.x): sendAllSessions=false sendAllSessionsSize=500 sendAllSessionsWait=20 What has me stumped is why the time required to do more sessions is exponentially higher beyond 1500 sessions. Using JMeter I can simulate 3600 new users (all creating a session) and the two servers can serve the requests AND generate/replicate the sessions in under 19 seconds. Any ideas would be greatly appreciated. I have a full test environment to simulate anything you might recommend. Sincerely, Kyle Harper From: Igor Cicimov icici...@gmail.com To: Tomcat Users List users@tomcat.apache.org Date: 09/05/2012 07:12 PM Subject:Re: Tuning session replication on clusters On Thu, Sep 6, 2012 at 5:51 AM, llow...@oreillyauto.com wrote: I have a small cluster of 3 nodes running tomcat 6.0.24 with openJDK 1.6.0_20 on Ubuntu 10.04 LTS. I have roughly 5,000-6,000 sessions at any given time, and when I restart one of the nodes I am finding that not all sessions are getting replicated , even when I have the state transfer timeout set to 60 seconds. It seems that only sessions that have been touched recently are replicated, even if the session is still otherwise valid. I did one test where I created about 1,500 sessions and then took out one node, When I brought it back online, it only replicated the 4-5 sessions that were from active users on the test cluster. It did not replicated the idle sessions that were still valid that my prior test had created. I am wanting to tune my settings, but I am unsure where would be the best place to start. Should I start with the threads available to the NIO Receiver, or would I be better off focusing on a different set of attributes first, such as the send or receive timeout values? Any tips or pointers as to which setting might be the most productive would be greatly appreciated. Lee Lowder O'Reilly Auto Parts Web Systems Administrator (417) 862-2674 x1858 This communication and any attachments are confidential, protected
Re: Tuning session replication on clusters
Alright, I did some more testing with another application and found the following: SessTime (sec 10 0.101 125 0.101 500 0.201 15000.201 18000.101 24000.101 42,000 0.901 (that's not a typo) Turns out the application that was having trouble is storing a silly amount of crap in the session. I am still not 100% sure what's happening behind the scenes at the 1500 session mark, but I'm guessing that based on our session size (nearly 700 MB) and memory configuration we were hitting some heap ceiling and the replication was forced to 'juggle'. If anyone has any more background on what's happening feel free to set me straight. I'll check back later... I need to go beat some developers... Kyle Harper From: kharp...@oreillyauto.com To: Tomcat Users List users@tomcat.apache.org Date: 09/05/2012 07:55 PM Subject:Re: Tuning session replication on clusters I'm working with Lee on this as well, so I can help answer most of that. In short: Yes, all our replication is working well. We have keepalived acting as a vrrp device (no round-robin dns) in front of a few web servers (apache 2.2.x, mod_proxy/mod_ajp) which are using stickysessions and BalancerMembers. Replication (DeltaManager/SimpleTCPCluster) is working as intended on the tomcat side (6.0.24). After further research, the problem we're seeing is performance with replication when the number of sessions is larger than around 2000. Using Jmeter on our test servers I can reproduce the problem. Here are the times it takes to replicate X number of sessions when an application is restarted: Sess Time (sec) 10 0.101 125 0.401 500 1.302 1500 2.104 1800 5.308 1800 6.709 2400 15.02 3600 30.285 3600 27.238 The times make sense until around 1500. The time it takes to replicate more than 1500 sessions becomes exponentially worse. Here is our cluster configuration from node1: Engine name=Catalina defaultHost=localhost jvmRoute=tntest-app-a-1 Cluster className=org.apache.catalina.ha.tcp.SimpleTcpCluster channelSendOptions=8 Manager className=org.apache.catalina.ha.session.DeltaManager stateTransferTimeout=45 expireSessionsOnShutdown=false notifyListenersOnReplication=true / Channel className=org.apache.catalina.tribes.group.GroupChannel Membership className=org.apache.catalina.tribes.membership.McastService address=239.255.0.1 port=45564 frequency=500 dropTime=3000 / Receiver className=org.apache.catalina.tribes.transport.nio.NioReceiver address=auto port=4000 autoBind=100 selectorTimeout=5000 maxThreads=6 / Sender className=org.apache.catalina.tribes.transport.ReplicationTransmitter Transport className=org.apache.catalina.tribes.transport.nio.PooledParallelSender timeout=45000 / /Sender Interceptor className=org.apache.catalina.tribes.group.interceptors.TcpFailureDetector/ Interceptor className=org.apache.catalina.tribes.group.interceptors.MessageDispatch15Interceptor/ /Channel Valve className=org.apache.catalina.ha.tcp.ReplicationValve filter=/ Valve className=org.apache.catalina.ha.session.JvmRouteBinderValve/ ClusterListener className=org.apache.catalina.ha.session.JvmRouteSessionIDBinderListener/ ClusterListener className=org.apache.catalina.ha.session.ClusterSessionListener/ /Cluster The best time we got for 3600 sessions was 24 seconds, and that's when I added the following to the Manager tag (stole this from the 5.5 docs; not even sure it's valid in 6.x): sendAllSessions=false sendAllSessionsSize=500 sendAllSessionsWait=20 What has me stumped is why the time required to do more sessions is exponentially higher beyond 1500 sessions. Using JMeter I can simulate 3600 new users (all creating a session) and the two servers can serve the requests AND generate/replicate the sessions in under 19 seconds. Any ideas would be greatly appreciated. I have a full test environment to simulate anything you might recommend. Sincerely, Kyle Harper From:Igor Cicimov icici...@gmail.com To: Tomcat Users List users@tomcat.apache.org Date:09/05/2012 07:12 PM Subject: Re: Tuning session replication on clusters On Thu, Sep 6, 2012 at 5:51 AM, llow...@oreillyauto.com wrote: I have a small cluster of 3 nodes running tomcat 6.0.24 with openJDK 1.6.0_20 on Ubuntu 10.04 LTS. I have roughly 5,000-6,000 sessions at any given time, and when I restart one of the nodes I am finding that not all sessions are getting
Re: Tuning session replication on clusters
On Thu, Sep 6, 2012 at 11:59 AM, kharp...@oreillyauto.com wrote: Alright, I did some more testing with another application and found the following: SessTime (sec 10 0.101 125 0.101 500 0.201 15000.201 18000.101 24000.101 42,000 0.901 (that's not a typo) Turns out the application that was having trouble is storing a silly amount of crap in the session. I am still not 100% sure what's happening behind the scenes at the 1500 session mark, but I'm guessing that based on our session size (nearly 700 MB) and memory configuration we were hitting some heap ceiling and the replication was forced to 'juggle'. If anyone has any more background on what's happening feel free to set me straight. I'll check back later... I need to go beat some developers... Kyle Harper From: kharp...@oreillyauto.com To: Tomcat Users List users@tomcat.apache.org Date: 09/05/2012 07:55 PM Subject:Re: Tuning session replication on clusters I'm working with Lee on this as well, so I can help answer most of that. In short: Yes, all our replication is working well. We have keepalived acting as a vrrp device (no round-robin dns) in front of a few web servers (apache 2.2.x, mod_proxy/mod_ajp) which are using stickysessions and BalancerMembers. Replication (DeltaManager/SimpleTCPCluster) is working as intended on the tomcat side (6.0.24). After further research, the problem we're seeing is performance with replication when the number of sessions is larger than around 2000. Using Jmeter on our test servers I can reproduce the problem. Here are the times it takes to replicate X number of sessions when an application is restarted: Sess Time (sec) 10 0.101 125 0.401 500 1.302 1500 2.104 1800 5.308 1800 6.709 2400 15.02 3600 30.285 3600 27.238 The times make sense until around 1500. The time it takes to replicate more than 1500 sessions becomes exponentially worse. Here is our cluster configuration from node1: Engine name=Catalina defaultHost=localhost jvmRoute=tntest-app-a-1 Cluster className=org.apache.catalina.ha.tcp.SimpleTcpCluster channelSendOptions=8 Manager className=org.apache.catalina.ha.session.DeltaManager stateTransferTimeout=45 expireSessionsOnShutdown=false notifyListenersOnReplication=true / Channel className=org.apache.catalina.tribes.group.GroupChannel Membership className=org.apache.catalina.tribes.membership.McastService address=239.255.0.1 port=45564 frequency=500 dropTime=3000 / Receiver className=org.apache.catalina.tribes.transport.nio.NioReceiver address=auto port=4000 autoBind=100 selectorTimeout=5000 maxThreads=6 / Sender className=org.apache.catalina.tribes.transport.ReplicationTransmitter Transport className=org.apache.catalina.tribes.transport.nio.PooledParallelSender timeout=45000 / /Sender Interceptor className=org.apache.catalina.tribes.group.interceptors.TcpFailureDetector/ Interceptor className=org.apache.catalina.tribes.group.interceptors.MessageDispatch15Interceptor/ /Channel Valve className=org.apache.catalina.ha.tcp.ReplicationValve filter=/ Valve className=org.apache.catalina.ha.session.JvmRouteBinderValve/ ClusterListener className=org.apache.catalina.ha.session.JvmRouteSessionIDBinderListener/ ClusterListener className=org.apache.catalina.ha.session.ClusterSessionListener/ /Cluster The best time we got for 3600 sessions was 24 seconds, and that's when I added the following to the Manager tag (stole this from the 5.5 docs; not even sure it's valid in 6.x): sendAllSessions=false sendAllSessionsSize=500 sendAllSessionsWait=20 What has me stumped is why the time required to do more sessions is exponentially higher beyond 1500 sessions. Using JMeter I can simulate 3600 new users (all creating a session) and the two servers can serve the requests AND generate/replicate the sessions in under 19 seconds. Any ideas would be greatly appreciated. I have a full test environment to simulate anything you might recommend. Maybe that's the boundary for the 6 threads used for the messages between the cluster members, having in mind the huge size of your sessions? By default the Sender uses 25 simultanious connections to each of the other nodes so maybe increasing this pool might speed up the things (poolSize value inside the Transport element of the Sender)? Sincerely, Kyle Harper From
