Re: Tuning session replication on clusters
Try jmap , jps,they will tell you about the usage of memory if you have doubt and monitor network data transfered 在 2012-9-6，13:59，kharp...@oreillyauto.com 写道： Alright, I did some more testing with another application and found the following: SessTime (sec 100.101 1250.101 5000.201 15000.201 18000.101 24000.101 42,0000.901 (that's not a typo) Turns out the application that was having trouble is storing a silly amount of crap in the session. I am still not 100% sure what's happening behind the scenes at the 1500 session mark, but I'm guessing that based on our session size (nearly 700 MB) and memory configuration we were hitting some heap ceiling and the replication was forced to 'juggle'. If anyone has any more background on what's happening feel free to set me straight. I'll check back later... I need to go beat some developers... Kyle Harper From:kharp...@oreillyauto.com To:Tomcat Users List users@tomcat.apache.org Date:09/05/2012 07:55 PM Subject:Re: Tuning session replication on clusters I'm working with Lee on this as well, so I can help answer most of that. In short: Yes, all our replication is working well. We have keepalived acting as a vrrp device (no round-robin dns) in front of a few web servers (apache 2.2.x, mod_proxy/mod_ajp) which are using stickysessions and BalancerMembers. Replication (DeltaManager/SimpleTCPCluster) is working as intended on the tomcat side (6.0.24). After further research, the problem we're seeing is performance with replication when the number of sessions is larger than around 2000. Using Jmeter on our test servers I can reproduce the problem. Here are the times it takes to replicate X number of sessions when an application is restarted: Sess Time (sec) 10 0.101 125 0.401 500 1.302 1500 2.104 1800 5.308 1800 6.709 2400 15.02 3600 30.285 3600 27.238 The times make sense until around 1500. The time it takes to replicate more than 1500 sessions becomes exponentially worse. Here is our cluster configuration from node1: Engine name=Catalina defaultHost=localhost jvmRoute=tntest-app-a-1 Cluster className=org.apache.catalina.ha.tcp.SimpleTcpCluster channelSendOptions=8 Manager className=org.apache.catalina.ha.session.DeltaManager stateTransferTimeout=45 expireSessionsOnShutdown=false notifyListenersOnReplication=true / Channel className=org.apache.catalina.tribes.group.GroupChannel Membership className=org.apache.catalina.tribes.membership.McastService address=239.255.0.1 port=45564 frequency=500 dropTime=3000 / Receiver className=org.apache.catalina.tribes.transport.nio.NioReceiver address=auto port=4000 autoBind=100 selectorTimeout=5000 maxThreads=6 / Sender className=org.apache.catalina.tribes.transport.ReplicationTransmitter Transport className=org.apache.catalina.tribes.transport.nio.PooledParallelSender timeout=45000 / /Sender Interceptor className=org.apache.catalina.tribes.group.interceptors.TcpFailureDetector/ Interceptor className=org.apache.catalina.tribes.group.interceptors.MessageDispatch15Interceptor/ /Channel Valve className=org.apache.catalina.ha.tcp.ReplicationValve filter=/ Valve className=org.apache.catalina.ha.session.JvmRouteBinderValve/ ClusterListener className=org.apache.catalina.ha.session.JvmRouteSessionIDBinderListener/ ClusterListener className=org.apache.catalina.ha.session.ClusterSessionListener/ /Cluster The best time we got for 3600 sessions was 24 seconds, and that's when I added the following to the Manager tag (stole this from the 5.5 docs; not even sure it's valid in 6.x): sendAllSessions=false sendAllSessionsSize=500 sendAllSessionsWait=20 What has me stumped is why the time required to do more sessions is exponentially higher beyond 1500 sessions. Using JMeter I can simulate 3600 new users (all creating a session) and the two servers can serve the requests AND generate/replicate the sessions in under 19 seconds. Any ideas would be greatly appreciated. I have a full test environment to simulate anything you might recommend. Sincerely, Kyle Harper From: Igor Cicimov icici...@gmail.com To: Tomcat Users List users@tomcat.apache.org Date: 09/05/2012 07:12 PM Subject: Re: Tuning session replication on clusters On Thu, Sep 6, 2012 at 5:51 AM, llow...@oreillyauto.com wrote: I have a small cluster of 3
RE: Facing Memory leak - 64 bit Tomcat 6.0.35 with windows 2008 R2(64 bit JVM 1.6.0_33)
Thanks a lot for your responses, I posted this query multiple times to get more and more responses. Please don’t consider this as spam, And after applying the answers(increase of memory) I got this issue resolved to some extent. Thanking you all again. Shailendra -Original Message- From: Christopher Schultz [mailto:ch...@christopherschultz.net] Sent: Wednesday, September 05, 2012 11:49 PM To: Tomcat Users List Subject: Re: Facing Memory leak - 64 bit Tomcat 6.0.35 with windows 2008 R2(64 bit JVM 1.6.0_33) -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Tim, On 9/5/12 1:17 PM, Tim Watts wrote: On Wed, 2012-09-05 at 12:16 -0400, PJ Delsh wrote: Shailendra, I'm not an expert, but when we had this same issue, we increased the Initial Memory Pool and Maximum Memory pool (XMS and XMX) in the Tomcat Monitor in Windows 2008.We also had a leak in some of our JSP files that was causing Tomcat to hang several times during business hours. We configured Tomcat so that VisualVM (which comes with the Java JDK) could monitor Tomcat memory usage. Then we took heap dumps, and analyzed them for leaks (Shallow Heap vs Retained Heap) using Eclipse Memory Analyzer. Once we fixed the leaks, Tomcat was stable.In the interim, if you have Tomcat running as a service in Windows, under the Recovery tab set the Tomcat service to restart automatically if it does stop. But if you have memory leaks, the Tomcat service probably will still be running but it won't be responding.You can also install more than one Tomcat on the same server, so if one goes down, the other will still be running. You would also have to configure Apache (or whatever) to work with more than one Tomcat.We also had issues with the Tomcat service crashing (eg Terminated Unsuccessfully) on Windows. After months of searching, we think the issue was having system.exit(0) in our code. System.exit(0) is the very definition of successful termination although it certainly doesn't belong in your webapp's code. +1 The truth is that Tomcat is not written well enough to run on Windows. What rot. All the problems you mention above you traced back to your application. Yet your conclusion is that Tomcat doesn't run well on Windows. Really? Tomcat runs very well on Microsoft Windows. Poorly-written applications will fail under any environment. In the OP's case, I really believe the only problem is that the heap is too small when moving from a 32-bit to a 64-bit JVM, but we can't seem to get s response from the OP so I guess we'll never know. See what you can do to move your app to Linux. You will find many more Tomcat experts on Linux than on Windows. I myself have a strong preference for Linux but for reasons unrelated to Tomcat. In reality, you'll find that most Tomcat experts don't give a rat's hiney which OS it runs under. But they may care which JVM you're using. I'm another Linux supporter and I'd hate to run Microsoft Windows as a server in general, but if you're stuck with it, Tomcat isn't going to be the problem. Tomcat is not like IIS. True. Developing for Tomcat on Windows is fine, but running production apps in Tomcat on Windows is a bad idea. I wish this was more widely known and publicized.-PJ Total BS. Agreed. PJ: care to cite any widely known and publicized references? Also, PJ, what makes you think that your experiences are anything like the OP's? You seem to be spouting solutions to problems that you do not understand. When the reported problem is we are getting OOMEs, the solution is not automatically you should switch to Linux because Tomcat sucks on Windows. It's inaccurate advice (Tomcat works find on Windows) and is very unlikely to solve the problem (whatever it is). - -chris -BEGIN PGP SIGNATURE- Version: GnuPG/MacGPG2 v2.0.17 (Darwin) Comment: GPGTools - http://gpgtools.org Comment: Using GnuPG with Mozilla - http://www.enigmail.net/ iEYEARECAAYFAlBHl4MACgkQ9CaO5/Lv0PAYmACdGQidM7rMSsG/+rZaA2P/zliB ibAAoIWe4z0Ifvy9h8WrYJdbZHvdc4Fp =9H6c -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tuning session replication on clusters
On Sep 5, 2012, at 8:48 PM, kharp...@oreillyauto.com wrote: I'm working with Lee on this as well, so I can help answer most of that. In short: Yes, all our replication is working well. We have keepalived acting as a vrrp device (no round-robin dns) in front of a few web servers (apache 2.2.x, mod_proxy/mod_ajp) which are using stickysessions and BalancerMembers. Replication (DeltaManager/SimpleTCPCluster) is working as intended on the tomcat side (6.0.24). Jumping in a little late on this thread, but have you considered trying the BackupManager instead of DeltaManager? The DeltaManager is going to replicate session data to all cluster members while BackupManager will only replicate to one backup cluster member. This might save you some time on restart. https://tomcat.apache.org/tomcat-7.0-doc/config/cluster-manager.html#org.apache.catalina.ha.session.BackupManager_Attributes Dan After further research, the problem we're seeing is performance with replication when the number of sessions is larger than around 2000. Using Jmeter on our test servers I can reproduce the problem. Here are the times it takes to replicate X number of sessions when an application is restarted: Sess Time (sec) 100.101 125 0.401 500 1.302 1500 2.104 1800 5.308 1800 6.709 2400 15.02 3600 30.285 3600 27.238 The times make sense until around 1500. The time it takes to replicate more than 1500 sessions becomes exponentially worse. Here is our cluster configuration from node1: Engine name=Catalina defaultHost=localhost jvmRoute=tntest-app-a-1 Cluster className=org.apache.catalina.ha.tcp.SimpleTcpCluster channelSendOptions=8 Manager className=org.apache.catalina.ha.session.DeltaManager stateTransferTimeout=45 expireSessionsOnShutdown=false notifyListenersOnReplication=true / Channel className=org.apache.catalina.tribes.group.GroupChannel Membership className=org.apache.catalina.tribes.membership.McastService address=239.255.0.1 port=45564 frequency=500 dropTime=3000 / Receiver className=org.apache.catalina.tribes.transport.nio.NioReceiver address=auto port=4000 autoBind=100 selectorTimeout=5000 maxThreads=6 / Sender className=org.apache.catalina.tribes.transport.ReplicationTransmitter Transport className=org.apache.catalina.tribes.transport.nio.PooledParallelSender timeout=45000 / /Sender Interceptor className=org.apache.catalina.tribes.group.interceptors.TcpFailureDetector/ Interceptor className=org.apache.catalina.tribes.group.interceptors.MessageDispatch15Interceptor/ /Channel Valve className=org.apache.catalina.ha.tcp.ReplicationValve filter=/ Valve className=org.apache.catalina.ha.session.JvmRouteBinderValve/ ClusterListener className=org.apache.catalina.ha.session.JvmRouteSessionIDBinderListener/ ClusterListener className=org.apache.catalina.ha.session.ClusterSessionListener/ /Cluster The best time we got for 3600 sessions was 24 seconds, and that's when I added the following to the Manager tag (stole this from the 5.5 docs; not even sure it's valid in 6.x): sendAllSessions=false sendAllSessionsSize=500 sendAllSessionsWait=20 What has me stumped is why the time required to do more sessions is exponentially higher beyond 1500 sessions. Using JMeter I can simulate 3600 new users (all creating a session) and the two servers can serve the requests AND generate/replicate the sessions in under 19 seconds. Any ideas would be greatly appreciated. I have a full test environment to simulate anything you might recommend. Sincerely, Kyle Harper From: Igor Cicimov icici...@gmail.com To: Tomcat Users List users@tomcat.apache.org Date: 09/05/2012 07:12 PM Subject: Re: Tuning session replication on clusters On Thu, Sep 6, 2012 at 5:51 AM, llow...@oreillyauto.com wrote: I have a small cluster of 3 nodes running tomcat 6.0.24 with openJDK 1.6.0_20 on Ubuntu 10.04 LTS. I have roughly 5,000-6,000 sessions at any given time, and when I restart one of the nodes I am finding that not all sessions are getting replicated , even when I have the state transfer timeout set to 60 seconds. It seems that only sessions that have been touched recently are replicated, even if the session is still otherwise valid. I did one test where I created about 1,500 sessions and then took out one node, When I brought it back online, it only replicated the 4-5 sessions that were from active users on the test cluster. It did not replicated the idle sessions that were still valid that my
org/eclipse/jdt/internal methods in tomcat 7 profilinig
Hi all, I am trying to profile tomcat 7 with java profiler. I start the tomacat run web application(small) in browser and stop the server. When I stop the server, I get the output from the profiler. Please note that I am using ubunut 12.04. I have put war file(chat.war) in webapps dir of tomcat. My confusion is here: When I see the profiler output, I see some methods of org/eclipse/jdt/internal... classes. *So where do this eclipse/jdt classes reside in tomcat ? What are they used for ? why each time they get executed when I run any web application ?** * I am new to tomcat and will appreciate your any help in finding answers to this questions. Thanks. Ragini.
Re: org/eclipse/jdt/internal methods in tomcat 7 profilinig
On 06/09/2012 13:19, Ragini wrote: So where do this eclipse/jdt classes reside in tomcat ? $CATALINA_HOME/lib/ecj-3.7.2.jar What are they used for ? Compiling the .java files generated from .jsp files into .class files. why each time they get executed when I run any web application ? They don't. They will get used every time you access a JSP that has not yet been compiled into a servlet. Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tuning session replication on clusters
Thanks all for replies (and for the jmap/jps idea, hadn't thought of that for some reason). I tried increasing the maxThreads on the NioReceviever and noticed no performance gain. I then modified the poolSize on the Transport element to 100 and saw no performance gain. This actually didn't surprise me after I discovered how large the sessions were. Using JMX (VisualVM) I watched the Heap size on my two servers as I tested 7000 sessions. Heap climbed approximately 1GB. When I restarted node2, I watched node1's heap usage nearly double. This confirmed my suspicion that the replication process is putting a copy of all sessions into a new object (list I suppose?) before transmitting them. After replication finished (109 seconds), node1's heap usage went back to normal. The aggregation of sessions into a new object to be sent (I presume as part of the handleGET_ALL_SESSIONS?) seems to work quickly, though I'm not sure how to test how much of the 109 seconds it took to replicate was Tomcat gathering up all the sessions to send and how much was network traffic. We have a low utilization gigabit ethernet fabric connecting all servers, so transferring 1GB of data shouldn't take more than 10-12 seconds. Does anyone know if there are ways to time the different steps in the replication process? If it is the network send/receive process that's slow, are there transmit/receive settings for the sender/receiver that could aid in speeding up replication of large data chunks? I see there are rxBufSize and txBufSize settings on the Receiver and Transport elements, and they're set to 25/43kb. If those values represents how data is chunked then larger settings might help (similar to the throughput difference of transferring 100x 10MB files vs. 10,000x 100kb files on a network). Any feedback is always appreciated. I will be able to test any suggestions later tonight and I'll be sure to report back if we make some headway. I'm sure this topic would be useful to others in the future. Thanks, Kyle Harper What has me stumped is why the time required to do more sessions is exponentially higher beyond 1500 sessions. Using JMeter I can simulate 3600 new users (all creating a session) and the two servers can serve the requests AND generate/replicate the sessions in under 19 seconds. Any ideas would be greatly appreciated. I have a full test environment to simulate anything you might recommend. Maybe that's the boundary for the 6 threads used for the messages between the cluster members, having in mind the huge size of your sessions? By default the Sender uses 25 simultanious connections to each of the other nodes so maybe increasing this pool might speed up the things (poolSize value inside the Transport element of the Sender)? This communication and any attachments are confidential, protected by Communications Privacy Act 18 USCS § 2510, solely for the use of the intended recipient, and may contain legally privileged material. If you are not the intended recipient, please return or destroy it immediately. Thank you. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
[ANN] Apache Tomcat 7.0.30 released
The Apache Tomcat team announces the immediate availability of Apache Tomcat 7.0.30. Apache Tomcat is an open source software implementation of the Java Servlet and JavaServer Pages technologies. This release contains numerous bug fixes and improvements compared to version 7.0.29. The notable changes include: - Significantly reduced memory footprint during web application start while Servlet 3.0 annotation and SCI scanning is in progress. - Adds support for scanning of classes that use Java 7 specific byte code for Servlet 3.0 annotation and SCI scanning. - Improvements to DIGEST and FORM authentication. Please refer to the change log for the complete list of changes: http://tomcat.apache.org/tomcat-7.0-doc/changelog.html Note: This version has 4 zip binaries: a generic one and three bundled with Tomcat native binaries for Windows operating systems running on different CPU architectures. Note: If you use the APR/native AJP or HTTP connector you *must* upgrade to version 1.1.24 or later of the AJP/native library Downloads: http://tomcat.apache.org/download-70.cgi Migration guides from Apache Tomcat 5.5.x and 6.0.x: http://tomcat.apache.org/migration.html Thank you, -- The Apache Tomcat Team - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat HeapMemoryUsage MBean question
Hi Christopher,
On Wed, Sep 5, 2012 at 9:46 AM, Christopher Schultz
ch...@christopherschultz.net wrote:
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
Shanti,
On 9/4/12 3:18 PM, Shanti Suresh wrote:
I would like to graph Tomcat's HeapMemoryUsage - used mbean value
for trending. I have installed Tomcat's manager application with a
user belonging only to the manager-jmx role. I would like to use
curl to get the data. I have a couple of questions:
(1) I don't seem to be able to assign an empty password or
otherwise no password to a user in tomcat-users.xml. Or could
I?
What did you try? What happened when you tried that?
I still get the basic authentication dialog box. And typing an empty
password, gives me an Unauthorized exception.
(2) How do I get just the used value from the composite
HeapMemoryUsage mbean? I guess I could subsequently search for
pattern used=. But just curious if I could get the value
directly. None of the following queries worked:
https://localhost:8453/manager/jmxproxy?qry=java.lang:type=Memory/HeapMemoryUsage
https://localhost:8453/manager/jmxproxy?qry=java.lang:type=Memory/HeapMemoryUsage:used
https://localhost:8453/manager/jmxproxy?qry=java.lang:type=Memory/HeapMemoryUsage.used
The
exact question you are asking is answered in an example of the
JMXProxy's get operation usage:
http://tomcat.apache.org/tomcat-7.0-doc/manager-howto.html#JMX_Get_command
Oh my! Thank you! I remember reading that a while back but just missed it
when in need. However, the syntax didn't produce the desired results.
https://hostname_fqdn:8453/manager/jmxproxy/?get=java.lang:type=Memoryatt=HeapMemoryUsagekey=used;
== gives me the composite result back:
OK - Attribute get 'java.lang:type=Memory' - HeapMemoryUsage=
javax.management.openmbean.CompositeDataSupport(compositeType=javax.management.openmbean.CompositeType(name=java.lang.management.MemoryUsage,items=((itemName=committed,itemType=javax.management.openmbean.SimpleType(name=java.lang.Long)),(itemName=init,itemType=javax.management.openmbean.SimpleType(name=java.lang.Long)),(itemName=max,itemType=javax.management.openmbean.SimpleType(name=java.lang.Long)),(itemName=used,itemType=javax.management.openmbean.SimpleType(name=java.lang.Long,contents={committed=422940672,
init=0, max=3087007744, used=116903640})
I will try some more. But I may be doing something wrong, perhaps?
Thanks.
-Shanti
- -chris
-BEGIN PGP SIGNATURE-
Version: GnuPG/MacGPG2 v2.0.17 (Darwin)
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Mozilla - http://www.enigmail.net/
iEYEARECAAYFAlBHV6EACgkQ9CaO5/Lv0PBvawCfS0UPId5r3QHubKNI81p+0Uo9
oigAnA6SZdrc7uli5looMShWeHv1NuoH
=TdSs
-END PGP SIGNATURE-
-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tuning session replication on clusters
On 06.09.2012 15:10, kharp...@oreillyauto.com wrote: ... This actually didn't surprise me after I discovered how large the sessions were. Using JMX (VisualVM) I watched the Heap size on my two servers as I tested 7000 sessions. Heap climbed approximately 1GB. When I restarted node2, I watched node1's heap usage nearly double. This confirmed my suspicion that the replication process is putting a copy of all sessions into a new object (list I suppose?) before transmitting them. After replication finished (109 seconds), node1's heap usage went back to normal. That's a plausible explanation for your observation. You can split replication in several chunks using the config items you already observed. Even in TC 6 the DeltaManager supports: sendAllSessions (Default: true, means all session send in one message, false means split in multiple messages) sendAllSessionsSize (Default: 1000, number of sessions send per message when switch is false) sendAllSessionsWaitTime (Default: 2000; sleep pause between sending consecutive messages) The aggregation of sessions into a new object to be sent (I presume as part of the handleGET_ALL_SESSIONS?) seems to work quickly, though I'm not sure how to test how much of the 109 seconds it took to replicate was Tomcat gathering up all the sessions to send and how much was network traffic. We have a low utilization gigabit ethernet fabric connecting all servers, so transferring 1GB of data shouldn't take more than 10-12 seconds. Does anyone know if there are ways to time the different steps in the replication process? Set log level of org.apache.catalina.ha.session.DeltaManager to DEBUG or FINE depending whether you are using log4j or juli for Tomcat. If it is the network send/receive process that's slow, Try sniffing both ends for network analysis. are there transmit/receive settings for the sender/receiver that could aid in speeding up replication of large data chunks? I see there are rxBufSize and txBufSize settings on the Receiver and Transport elements, and they're set to 25/43kb. If those values represents how data is chunked then larger settings might help (similar to the throughput difference of transferring 100x 10MB files vs. 10,000x 100kb files on a network). Regards, Rainer - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: When will be the tomcat version 7.0.30 released
On 06.09.2012 16:56, Sunny Mittal wrote: I upgraded to tomcat 7.0.29 version and found that it has some Out of Memory issues. So we are planning to wait and upgrade to 7.0.30. Can you tell what is the release date for tomcat 7.0.30? Current expectation is between hours and very few days. Regards, Rainer - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tuning session replication on clusters
On 06.09.2012 16:57, Rainer Jung wrote: On 06.09.2012 15:10, kharp...@oreillyauto.com wrote: ... This actually didn't surprise me after I discovered how large the sessions were. Using JMX (VisualVM) I watched the Heap size on my two servers as I tested 7000 sessions. Heap climbed approximately 1GB. When I restarted node2, I watched node1's heap usage nearly double. This confirmed my suspicion that the replication process is putting a copy of all sessions into a new object (list I suppose?) before transmitting them. After replication finished (109 seconds), node1's heap usage went back to normal. That's a plausible explanation for your observation. You can split replication in several chunks using the config items you already observed. Even in TC 6 the DeltaManager supports: sendAllSessions (Default: true, means all session send in one message, false means split in multiple messages) sendAllSessionsSize (Default: 1000, number of sessions send per message when switch is false) sendAllSessionsWaitTime (Default: 2000; sleep pause between sending consecutive messages) I forgot one more thing: since TC 6.0.34 and 7.0.22 is it possible to decide which session attributes get replicated. So in case you have only few attributes that make up most of the big session memory *and* your application is able to transparently handle the situation, that these attributes are suddenly missing from the session, e.g. by retrieving the data again from some back end system or database, the following might help: Look for sessionAttributeFilter in http://tomcat.apache.org/tomcat-6.0-doc/config/cluster-manager.html I'm not saying it is easy, but if you want to make your application using session replication really efficient, it is a possible way to go. In addition there is a way an application can detect whether there was a node fail over, ie. a request is handled by another node as the previous request for the same session. You can hook filling missing attributes on this detection. The detection uses a feature of the ReplicationValve, which can set a request attribute that can be inspected to decide whether there was a fail over. Look for primaryIndicator in http://tomcat.apache.org/tomcat-6.0-doc/config/cluster-valve.html. If the attribute is false, you just switched nodes (fail over) and are now working on a replicated session. Regards, Rainer - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: When will be the tomcat version 7.0.30 released
On 06.09.2012 17:01, Rainer Jung wrote: On 06.09.2012 16:56, Sunny Mittal wrote: I upgraded to tomcat 7.0.29 version and found that it has some Out of Memory issues. So we are planning to wait and upgrade to 7.0.30. Can you tell what is the release date for tomcat 7.0.30? Current expectation is between hours and very few days. Correct to self: it *was* already released about an hour ago. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tuning session replication on clusters
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Kyle, On 9/5/12 9:59 PM, kharp...@oreillyauto.com wrote: Alright, I did some more testing with another application and found the following: SessTime (sec 10 0.101 125 0.101 500 0.201 1500 0.201 1800 0.101 24000.101 42,0000.901 (that's not a typo) Good to know that Tomcat itself does not seem to be the problem: thanks for the report. One question: After further research, the problem we're seeing is performance with replication when the number of sessions is larger than around 2000. Using Jmeter on our test servers I can reproduce the problem. Here are the times it takes to replicate X number of sessions when an application is restarted: Sess Time (sec) 10 0.101 125 0.401 500 1.302 1500 2.104 1800 5.308 1800 6.709 2400 15.02 3600 30.285 3600 27.238 The times make sense until around 1500. The time it takes to replicate more than 1500 sessions becomes exponentially worse. Plot those as X-Y in a spreadsheet and you'll see that it's only a bit worse than linear, especially after 1500. There's no enough data presented to draw an exponential performance curve conclusion: you're going to need more data to see if this is worse than linear. Can you collect more data? I know that creating 42k sessions of 700MiB each is probably a .. challenge, but more data would certainly be helpful. - -chris -BEGIN PGP SIGNATURE- Version: GnuPG/MacGPG2 v2.0.17 (Darwin) Comment: GPGTools - http://gpgtools.org Comment: Using GnuPG with Mozilla - http://www.enigmail.net/ iEYEARECAAYFAlBIwxkACgkQ9CaO5/Lv0PCrxwCeLMDD9Zx7PW6sdhnF+K+ONP79 HP0AoLfdb6NTm8xGHRimYqdqap1oQKTX =x6Z6 -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: When will be the tomcat version 7.0.30 released
Den 06-09-2012 17:16, Rainer Jung skrev: On 06.09.2012 17:01, Rainer Jung wrote: On 06.09.2012 16:56, Sunny Mittal wrote: I upgraded to tomcat 7.0.29 version and found that it has some Out of Memory issues. So we are planning to wait and upgrade to 7.0.30. Can you tell what is the release date for tomcat 7.0.30? Current expectation is between hours and very few days. Correct to self: it *was* already released about an hour ago. Also I don't think tomcat has any OOME's, if anything it is the webapp(s) running inside tomcat that needs more memory so just give it that? - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Tomcat running with a shared unix group but unable to read files with group permissions
Hi all,
I'm stumped on a seemingly java/tomcat related issue and am hoping someone
can provide some help.
We have two users ('user1' and 'user2') on our linux server that share the
same group ('group1'). User 'user1' writes some files that have the
following permissions:
-rw-r- 1 user1 group1 788 Sep 5 19:42 file.log
The folder containing this file has the following permissions:
drwxr-xr-- 2 user1 group1 4096 Sep 5 19:42 log
The tomcat web app is launched as user 'user2'. Below is the ps output for
the process. I've also verified that the java web app is running with gid
of the shared group 'group1'.
user231920 31919 99 21:30 ?00:00:36 /usr/local/jre/bin/java
org.apache.catalina.startup.Bootstrap start
When the web app tries to read the file, *it gets the following exception*:
java.io.FileNotFoundException: /foo/bar/data/log/file.log (Permission
denied)
at java.io.RandomAccessFile.open(Native Method)
at java.io.RandomAccessFile.init(RandomAccessFile.java:233)
at java.io.RandomAccessFile.init(RandomAccessFile.java:118)
…
at java.lang.Thread.run(Thread.java:679)
However, while logged in as 'user2', I can run a simple
cat /foo/bar/data/log/file.log and* I can read the contents of the file*.
Also, if I provide 'other' read permissions to the file (e.g. -rw-r--r--
1 user1 group1 788 Sep 5 19:42 file.log), *the web app is able to read
the file*.
If I write a sample java application that tries to read this file and
execute it while logged in as 'user2', again *Java is able to read the file.
*
Tomcat doesn't seem to be using any security policy as far as I can tell.
Any ideas why the group permissions seem to be ignored by tomcat?
Thanks!
Udam
Re: Tuning session replication on clusters
Rainer: Thanks for the input. I'll do some additional testing with the sendAllSessions attributes, but my initial testing didn't show much gain. If the rx/tx settings are already chunking up the session bytes into smallish payloads then nothing I change with the sendAllSessions will improve that. I might be completely off base, but I'm unsure of what the rxBufSize and txBufSize attributes do exactly, and if changing them would boost throughput. Chris: One question: Plot those as X-Y in a spreadsheet and you'll see that it's only a bit worse than linear, especially after 1500. There's no enough data presented to draw an exponential performance curve conclusion: you're going to need more data to see if this is worse than linear. Can you collect more data? I know that creating 42k sessions of 700MiB each is probably a .. challenge, but more data would certainly be helpful. Before I change anything (tuning) I'll do as much testing as I can tonight to generate a data series and graph. Btw, does this list support .ods/.xls attachments? I'm not sure how Tomcat handles building the object containing all the sessions to send (Collection/List?), but it'll be interesting to see if there are points where the basic nature of the Collection object becomes less efficient. The testing so far has already shown that the quantity of sessions (like the 42k test I did in 901 ms on a different app) doesn't appear to be an issue, while large session sizes do. I know that the JVM was forced to make a full copy of all the session data for the application inside heap space too, so after certain points I might be reaching contention with memory management processes that handle heap size, ratios, garbage collection, etc. Having seen how quickly the JVM ramped up memory usage for replication, I'm relatively confident the session aggregation and duplication into a new Collection on the sender side is happening fast... realy fast. I'll try to get a protocol analyzer in place to capture the packets and compare them to a basic file transfer of a single large file. If anyone knows much about those rxBufSize and txBufSize settings and whether they'll impact anything let me know. If they operate like TCP congestion and window sizes then adjusting them could have profound performance implications. Thanks! Kyle Harper This communication and any attachments are confidential, protected by Communications Privacy Act 18 USCS § 2510, solely for the use of the intended recipient, and may contain legally privileged material. If you are not the intended recipient, please return or destroy it immediately. Thank you. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat running with a shared unix group but unable to read files with group permissions
Udam Dewaraja wrote:
Hi all,
I'm stumped on a seemingly java/tomcat related issue and am hoping someone
can provide some help.
We have two users ('user1' and 'user2') on our linux server that share the
same group ('group1'). User 'user1' writes some files that have the
following permissions:
-rw-r- 1 user1 group1 788 Sep 5 19:42 file.log
The folder containing this file has the following permissions:
drwxr-xr-- 2 user1 group1 4096 Sep 5 19:42 log
The tomcat web app is launched as user 'user2'. Below is the ps output for
the process. I've also verified that the java web app is running with gid
of the shared group 'group1'.
user231920 31919 99 21:30 ?00:00:36 /usr/local/jre/bin/java
org.apache.catalina.startup.Bootstrap start
When the web app tries to read the file, *it gets the following exception*:
java.io.FileNotFoundException: /foo/bar/data/log/file.log (Permission
denied)
at java.io.RandomAccessFile.open(Native Method)
at java.io.RandomAccessFile.init(RandomAccessFile.java:233)
at java.io.RandomAccessFile.init(RandomAccessFile.java:118)
…
at java.lang.Thread.run(Thread.java:679)
However, while logged in as 'user2', I can run a simple
cat /foo/bar/data/log/file.log and* I can read the contents of the file*.
Also, if I provide 'other' read permissions to the file (e.g. -rw-r--r--
1 user1 group1 788 Sep 5 19:42 file.log), *the web app is able to read
the file*.
If I write a sample java application that tries to read this file and
execute it while logged in as 'user2', again *Java is able to read the file.
*
Tomcat doesn't seem to be using any security policy as far as I can tell.
Any ideas why the group permissions seem to be ignored by tomcat?
Nothing to do with Tomcat I think.
Maybe it is because java.io.RandomAccessFile is a read/write kind of file, and the group
just has read permission ?
All your tests involve reading, not writing, and reading is allowed for the
group.
Google for java.io.RandomAccessFile.
-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat running with a shared unix group but unable to read files with group permissions
In my code, the RandomAccess file is trying to do a read (code below).
That's why all my tests are doing reads.
logFile = new RandomAccessFile(fileToRead, r);
The sample java application I ran executes the exact same line above (with
the same file) and reads the contents correctly. However, in Tomcat webapp,
this fails.
Thanks,
Udam
On Thu, Sep 6, 2012 at 1:15 PM, André Warnier a...@ice-sa.com wrote:
Udam Dewaraja wrote:
Hi all,
I'm stumped on a seemingly java/tomcat related issue and am hoping someone
can provide some help.
We have two users ('user1' and 'user2') on our linux server that share the
same group ('group1'). User 'user1' writes some files that have the
following permissions:
-rw-r- 1 user1 group1 788 Sep 5 19:42 file.log
The folder containing this file has the following permissions:
drwxr-xr-- 2 user1 group1 4096 Sep 5 19:42 log
The tomcat web app is launched as user 'user2'. Below is the ps output for
the process. I've also verified that the java web app is running with gid
of the shared group 'group1'.
user231920 31919 99 21:30 ?00:00:36 /usr/local/jre/bin/java
org.apache.catalina.startup.**Bootstrap start
When the web app tries to read the file, *it gets the following
exception*:
java.io.FileNotFoundException: /foo/bar/data/log/file.log (Permission
denied)
at java.io.RandomAccessFile.open(**Native Method)
at java.io.RandomAccessFile.**init(RandomAccessFile.java:**233)
at java.io.RandomAccessFile.**init(RandomAccessFile.java:**118)
…
at java.lang.Thread.run(Thread.**java:679)
However, while logged in as 'user2', I can run a simple
cat /foo/bar/data/log/file.log and* I can read the contents of the file*.
Also, if I provide 'other' read permissions to the file (e.g. -rw-r--r--
1 user1 group1 788 Sep 5 19:42 file.log), *the web app is able to read
the file*.
If I write a sample java application that tries to read this file and
execute it while logged in as 'user2', again *Java is able to read the
file.
*
Tomcat doesn't seem to be using any security policy as far as I can tell.
Any ideas why the group permissions seem to be ignored by tomcat?
Nothing to do with Tomcat I think.
Maybe it is because java.io.RandomAccessFile is a read/write kind of file,
and the group just has read permission ?
All your tests involve reading, not writing, and reading is allowed for
the group.
Google for java.io.RandomAccessFile.
--**--**-
To unsubscribe, e-mail:
users-unsubscribe@tomcat.**apache.orgusers-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tuning session replication on clusters
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Kyle, On 9/6/12 2:45 PM, kharp...@oreillyauto.com wrote: Chris: One question: Plot those as X-Y in a spreadsheet and you'll see that it's only a bit worse than linear, especially after 1500. There's no enough data presented to draw an exponential performance curve conclusion: you're going to need more data to see if this is worse than linear. Can you collect more data? I know that creating 42k sessions of 700MiB each is probably a .. challenge, but more data would certainly be helpful. Before I change anything (tuning) I'll do as much testing as I can tonight to generate a data series and graph. Btw, does this list support .ods/.xls attachments? I'm not sure. You might be better off wrapping the file in a ZIP file (which is silly, since OpenDocument is already ZIP, but it might help) or just throwing the file up on pastebin (which I generally don't like because the message goes into the archives but pastebin items expire, etc. Just try not to drop a multi-MiB file onto the list ;) I'm not sure how Tomcat handles building the object containing all the sessions to send (Collection/List?), but it'll be interesting to see if there are points where the basic nature of the Collection object becomes less efficient. The testing so far has already shown that the quantity of sessions (like the 42k test I did in 901 ms on a different app) doesn't appear to be an issue, while large session sizes do. Assembling the sessions into a Collection is likely to be very fast, since it's just copying references around: the size of the individual sessions should not matter. Of course, pushing all those bytes to the other servers... I know that the JVM was forced to make a full copy of all the session data for the application inside heap space too, so after certain points I might be reaching contention with memory management processes that handle heap size, ratios, garbage collection, etc. Perhaps Tomcat does something like serialize the session to a big binary structure and then sends that (which sounds insane -- streaming binary data is how that should be done -- but I haven't checked to code to be sure). Having seen how quickly the JVM ramped up memory usage for replication, I'm relatively confident the session aggregation and duplication into a new Collection on the sender side is happening fast... realy fast. I'll try to get a protocol analyzer in place to capture the packets and compare them to a basic file transfer of a single large file. Sounds good. - -chris -BEGIN PGP SIGNATURE- Version: GnuPG/MacGPG2 v2.0.17 (Darwin) Comment: GPGTools - http://gpgtools.org Comment: Using GnuPG with Mozilla - http://www.enigmail.net/ iEYEARECAAYFAlBJI7YACgkQ9CaO5/Lv0PDcIwCfYaKt6ddWk3Nok6bxR2RX8ouX XEYAni+8N/oGPx/vl9GTwPfVXEek0u/Y =Tdtz -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat HeapMemoryUsage MBean question
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
Shanti,
On 9/6/12 10:15 AM, Shanti Suresh wrote:
Oh my! Thank you! I remember reading that a while back but just
missed it when in need. However, the syntax didn't produce the
desired results.
https://hostname_fqdn:8453/manager/jmxproxy/?get=java.lang:type=Memoryatt=HeapMemoryUsagekey=used;
== gives me the composite result back:
OK - Attribute get 'java.lang:type=Memory' - HeapMemoryUsage=
javax.management.openmbean.CompositeDataSupport(compositeType=javax.management.openmbean.CompositeType(name=java.lang.management.MemoryUsage,items=((itemName=committed,itemType=javax.management.openmbean.SimpleType(name=java.lang.Long)),(itemName=init,itemType=javax.management.openmbean.SimpleType(name=java.lang.Long)),(itemName=max,itemType=javax.management.openmbean.SimpleType(name=java.lang.Long)),(itemName=used,itemType=javax.management.openmbean.SimpleType(name=java.lang.Long,contents={committed=422940672,
init=0, max=3087007744, used=116903640})
I will try some more. But I may be doing something wrong,
perhaps?
The key parameter was added in 7.0.27 and I don't believe it was
back-ported to Tomcat 6.0.x. What version are you running?
- -chris
-BEGIN PGP SIGNATURE-
Version: GnuPG/MacGPG2 v2.0.17 (Darwin)
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Mozilla - http://www.enigmail.net/
iEYEARECAAYFAlBJNkwACgkQ9CaO5/Lv0PBr/QCgru/OI/DxTEbHGXgiaVXHwFuM
9UkAoIYWd9oH+Hm0j+IaDtE+cDOWtr0s
=/mh1
-END PGP SIGNATURE-
-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
