Re: Help with Tomcat 7 clustering using BIO receiver
Hello Filip You solved my issue! Thank you very much! Thanks everyone João
Re: Help with Tomcat 7 clustering using BIO receiver
Joao, try channelSendOptions="6" this will mean that 1. You wish to use ACK's (option 2) 2. You wish the ACK to be synchronous If you don't have the 0x0002 option enabled, it wont use ACKs at all. Filip On Thu, Jul 3, 2014 at 4:44 PM, João Sávio wrote: > If I set channelSendOptions="8" (default value = asynchronous), the % of > errors increase (as expected) > > Regards > João > > > 2014-07-03 19:43 GMT-03:00 João Sávio : > > > I don't think so. > > > > Here it is: http://pastebin.com/qYCzmECb (server.xml - node1) > > > > Regards > > João > > > > > > -- > http://joaosavio.wordpress.com >
Re: Help with Tomcat 7 clustering using BIO receiver
If I set channelSendOptions="8" (default value = asynchronous), the % of errors increase (as expected) Regards João 2014-07-03 19:43 GMT-03:00 João Sávio : > I don't think so. > > Here it is: http://pastebin.com/qYCzmECb (server.xml - node1) > > Regards > João > -- http://joaosavio.wordpress.com
Re: Help with Tomcat 7 clustering using BIO receiver
I don't think so. Here it is: http://pastebin.com/qYCzmECb (server.xml - node1) Regards João
Re: Help with Tomcat 7 clustering using BIO receiver
did you post your server.xml cause I can't find it? On Thu, Jul 3, 2014 at 4:25 PM, João Sávio wrote: > Hello Filip > > I'm using channelSendOptions="4", which is supposed to be synchronous > > Regards > João >
Re: Help with Tomcat 7 clustering using BIO receiver
Hello Filip I'm using channelSendOptions="4", which is supposed to be synchronous Regards João
Re: Help with Tomcat 7 clustering using BIO receiver
A race condition could happen if you set replication to happen async. But I do have a memory of the configuration specifying synchronous replication, which would guarantee that the replication changes have happened before the request is complete. On Thu, Jul 3, 2014 at 3:51 PM, Christopher Schultz < ch...@christopherschultz.net> wrote: > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA256 > > Mark, > > On 7/3/14, 2:19 PM, Mark Eggers wrote: > > João, > > > > This list has a convention of posting either inline or at the end > > of the message you're replying to. > > > > See here for mailing list notes: > > > > http://tomcat.apache.org/lists.html#tomcat-users > > > > On 7/3/2014 10:24 AM, João Sávio wrote: > >> Hello > > > >> Some points below: > > > >> ** What is "on time"?* In my application, a group of users should > >> always hit the same node after the first request. So, in first > >> request each group of users will receive an specific cookie, and > >> LB will perform the load balancing based on this cookie. In first > >> request, a user can hit any node, but from the second, he or she > >> should hit the same node. > > > > Hmm, so 'on time' really means that subsequent requests should hit > > the same server. > > > > If you're using sessions, Tomcat has an attribute on the Engine > > element called jvmRoute. So depending on your load balancer (and > > if you use AJP), you can use Tomcat and AJP to route traffic. In > > that case, there's no need to write a special cookie. > > > > At any rate, this doesn't sound like a clustering error per se. > > > I wonder if the real problem is a race condition: the cluster can't > replicate fast enough to stabilize before the second request comes in, > plus the lb configuration might not be correct. > > João, can you confirm that request #1 and #2 are definitely hitting > the same Tomcat instance? > > If you connect to TomcatA, set a session attribute, then reconnect to > TomcatA and get that session attribute, then it should be the same > unless something is awfully wrong. You don't even need to have > clustering enabled to test the above. > > However if you hit TomcatA, set a session attribute, then connect to > TomcatB and try to get that session attribute, your request may have > arrived too early for the cluster to have pushed-out the session > attribute change. > > So, if you can prove that both requests have gone to TomcatA and you > are getting "errors", then there are several possibilities: > > 1. Tomcat has a huge bug and no web applications would work worldwide. > 2. You are mishandling the "setting" of the session attribute > 3. You are wrong about which server the client is hitting > > - -chris > -BEGIN PGP SIGNATURE- > Version: GnuPG v1 > Comment: GPGTools - http://gpgtools.org > Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ > > iQIcBAEBCAAGBQJTtdBTAAoJEBzwKT+lPKRYtiwP/1dW2qplyepTgDTNixNw0viZ > 29XFywsYAmDdMxzWcgkcl7Nrw3kVUcJVf+jLpxNCUxRJq7z4+zuyOLkImn2XW4a+ > ygG1op69FSwsVEfQyHIH8OVjdYDUj6WPpP8bu2KbbkR0jtAiHO569+869WOvPyuA > z+oBhBhWB5w7e41qmQnLr6y3+hU19hGuayxkR61tqmZCPp6kpwRH2yN3IbhId2In > 8DLoR5z6077jxPeXR6o3goB6Y9LbrPoYFUwdfQTpzrF8AvQ2wDl/CRuM2n9wB/ez > Oclnz0bw4JNegtarEJeiu4G1Qqf7WCqhVv4a8GfWYtr0ISk8GBBcCRjYZcoyU5IU > hSnNBGn586AhZ3BK5t1ySwrC6RiKH6MIR8fdBOSw1eZnTycPBSK6avZ4E8ahQDIp > uA93W+cME58gtmzdl2q7iLjRbwGdgebw++yfR4G42Tb4rUYsmOzsCPGx/nIqxB5E > FBea8xGwb802rFpYMxgMp8SzRy078RrDx2aptNfrb5oP9YeQ/pGrX9tVVtTlxNTk > 8DKA8GHL4fiONAJB48iD2sTSv4jAhFInHnF4ykl0zjN7t3f0phMmSExeoH7HbFUI > G589M4KAs5X00xCSFt9gXdU+tpuFL+/x6kBAGrNmT5IySIvm+BfxTXjvg2daAjcC > +FAocYeosZumP5g2tICv > =1Si4 > -END PGP SIGNATURE- > > - > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > >
Re: Help with Tomcat 7 clustering using BIO receiver
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Mark, On 7/3/14, 2:19 PM, Mark Eggers wrote: > João, > > This list has a convention of posting either inline or at the end > of the message you're replying to. > > See here for mailing list notes: > > http://tomcat.apache.org/lists.html#tomcat-users > > On 7/3/2014 10:24 AM, João Sávio wrote: >> Hello > >> Some points below: > >> ** What is "on time"?* In my application, a group of users should >> always hit the same node after the first request. So, in first >> request each group of users will receive an specific cookie, and >> LB will perform the load balancing based on this cookie. In first >> request, a user can hit any node, but from the second, he or she >> should hit the same node. > > Hmm, so 'on time' really means that subsequent requests should hit > the same server. > > If you're using sessions, Tomcat has an attribute on the Engine > element called jvmRoute. So depending on your load balancer (and > if you use AJP), you can use Tomcat and AJP to route traffic. In > that case, there's no need to write a special cookie. > > At any rate, this doesn't sound like a clustering error per se. I wonder if the real problem is a race condition: the cluster can't replicate fast enough to stabilize before the second request comes in, plus the lb configuration might not be correct. João, can you confirm that request #1 and #2 are definitely hitting the same Tomcat instance? If you connect to TomcatA, set a session attribute, then reconnect to TomcatA and get that session attribute, then it should be the same unless something is awfully wrong. You don't even need to have clustering enabled to test the above. However if you hit TomcatA, set a session attribute, then connect to TomcatB and try to get that session attribute, your request may have arrived too early for the cluster to have pushed-out the session attribute change. So, if you can prove that both requests have gone to TomcatA and you are getting "errors", then there are several possibilities: 1. Tomcat has a huge bug and no web applications would work worldwide. 2. You are mishandling the "setting" of the session attribute 3. You are wrong about which server the client is hitting - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1 Comment: GPGTools - http://gpgtools.org Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQIcBAEBCAAGBQJTtdBTAAoJEBzwKT+lPKRYtiwP/1dW2qplyepTgDTNixNw0viZ 29XFywsYAmDdMxzWcgkcl7Nrw3kVUcJVf+jLpxNCUxRJq7z4+zuyOLkImn2XW4a+ ygG1op69FSwsVEfQyHIH8OVjdYDUj6WPpP8bu2KbbkR0jtAiHO569+869WOvPyuA z+oBhBhWB5w7e41qmQnLr6y3+hU19hGuayxkR61tqmZCPp6kpwRH2yN3IbhId2In 8DLoR5z6077jxPeXR6o3goB6Y9LbrPoYFUwdfQTpzrF8AvQ2wDl/CRuM2n9wB/ez Oclnz0bw4JNegtarEJeiu4G1Qqf7WCqhVv4a8GfWYtr0ISk8GBBcCRjYZcoyU5IU hSnNBGn586AhZ3BK5t1ySwrC6RiKH6MIR8fdBOSw1eZnTycPBSK6avZ4E8ahQDIp uA93W+cME58gtmzdl2q7iLjRbwGdgebw++yfR4G42Tb4rUYsmOzsCPGx/nIqxB5E FBea8xGwb802rFpYMxgMp8SzRy078RrDx2aptNfrb5oP9YeQ/pGrX9tVVtTlxNTk 8DKA8GHL4fiONAJB48iD2sTSv4jAhFInHnF4ykl0zjN7t3f0phMmSExeoH7HbFUI G589M4KAs5X00xCSFt9gXdU+tpuFL+/x6kBAGrNmT5IySIvm+BfxTXjvg2daAjcC +FAocYeosZumP5g2tICv =1Si4 -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Help with Tomcat 7 clustering using BIO receiver
Ok, at least the stack trace is clear. The session has been invalidated somehow. We would need to figure out when and how this happens, is it possible that you are doing a clean shutdown of a tomcat instance and that instance expires all the sessions? If that is the case, kill the tomcat with 'kill -9' to simulate a failure. there is a flag called 'expireSessionsOnShutdown', do you have that set? On Thu, Jul 3, 2014 at 2:05 PM, João Sávio wrote: > Hello Mark > > In fact, I'm not explicit invalidating session on this two requests. > > Regards > João >
Re: Help with Tomcat 7 clustering using BIO receiver
Hello Mark In fact, I'm not explicit invalidating session on this two requests. Regards João
Re: Help with Tomcat 7 clustering using BIO receiver
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Filip, On 7/3/2014 12:11 PM, Filip Hanik wrote: > 1. are your machines in time sync? If they are not, a session can > get timed out. 2. 3. SEVERE: Manager [localhost#/myApp]: Unable to > receive message through TCP channel 4. > java.lang.IllegalStateException: setAttribute: Session [ > DEC3612CF763194E7953DB3FD2C433E0] has already been invalidated 5. > at org.apache.catalina.session.StandardSession.setAttribute( > StandardSession.java:1437) 6. at > org.apache.catalina.ha.session.DeltaSession.setAttribute( > DeltaSession.java:695) 7. at > org.apache.catalina.ha.session.DeltaRequest.execute( > DeltaRequest.java:168) 8. at > org.apache.catalina.ha.session.DeltaManager.handleSESSION_DELTA( > DeltaManager.java:1337) 9. at > org.apache.catalina.ha.session.DeltaManager.messageReceived > (DeltaManager.java:1283) 10. at > org.apache.catalina.ha.session.DeltaManager.messageDataReceived( > DeltaManager.java:1001) 11. at > org.apache.catalina.ha.session.ClusterSessionListener.messageReceived( > > ClusterSessionListener.java:91) > 12. at > org.apache.catalina.ha.tcp.SimpleTcpCluster.messageReceived( > SimpleTcpCluster.java:943) 13. at > org.apache.catalina.ha.tcp.SimpleTcpCluster.messageReceived( > SimpleTcpCluster.java:924) 14. at > org.apache.catalina.tribes.group.GroupChannel.messageReceived( > GroupChannel.java:278) > > I think he's running the Tomcat cluster on the same physical machine for testing. So is the web application invalidating the session before the attribute is replicated across the cluster? /mde/ > > > On Thu, Jul 3, 2014 at 1:07 PM, João Sávio > wrote: > >> Hi everyone >> >> I ran my test (total of 1k requests, total of 100 threads) >> against two nodes with default VM settings. I've just set heap >> size. I had about 15% of errors. >> >> cluster.log - node1 - http://pastebin.com/cpX900Qw cluster.log - >> node2 - http://pastebin.com/qCSzMaU6 >> >> Running for a long time (total of 500k requests, total of 100 >> threads) I had about 11% of errors. In this case we can see the >> statistics: >> >> Jul 03, 2014 5:53:28 PM >> org.apache.catalina.tribes.group.interceptors.ThroughputInterceptor >> report INFO: ThroughputInterceptor Report[ Tx Msg:1 messages >> Sent:12.36 MB (total) Sent:12.36 MB (application) Time:7.82 >> seconds Tx Speed:1.58 MB/sec (total) TxSpeed:1.58 MB/sec >> (application) Error Msg:0 Rx Msg:10198 messages Rx Speed:0.08 >> MB/sec (since 1st msg) Received:12.38 MB] >> >> >> All session attributes are Serializable, and it's a session >> replication issue because if I ran my test with just one node, I >> had 0% of errors. >> >> Regarding "on time", just a correction: >> >> 1. first request, pick a random server and store a session >> object 2. second request, pick *ANY* server (chose by LB based on >> the cookie - it can be the same, but not necessarily) and ask for >> the session object. >> >> To be more clean, I've been working with a conference system. >> Each conference should occur in one node. So, the first request >> can hit any server, and from the second request should hit the >> node where the conference is. >> >> >> Thanks a lot João >> >> >> 2014-07-03 15:40 GMT-03:00 Filip Hanik : >> >>> you mention NIO and say maxThreads, that sounds like the >>> configuration, but the BIO receiver is on the >>> cluster, and it a >> completely >>> different component that also has an applicable NIO >>> configuration. >>> >>> are you confusing the two? I'm saying that you should use the >>> NIO receiver on the cluster component, and if you do, what kind >>> of errors do you get? >>> >>> >>> On Thu, Jul 3, 2014 at 12:19 PM, Mark Eggers >> >>> >>> wrote: >>> > João, > > This list has a convention of posting either inline or at the end > of the message you're replying to. > > See here for mailing list notes: > > http://tomcat.apache.org/lists.html#tomcat-users > > On 7/3/2014 10:24 AM, João Sávio wrote: >> Hello >> >> Some points below: >> >> ** What is "on time"?* In my application, a group of >> users should always hit the same node after the first >> request. So, in first request each group of users will >> receive an specific cookie, and LB will perform the load >> balancing based on this cookie. In first request, a user >> can hit any node, but from the second, he or she should >> hit the same node. > > Hmm, so 'on time' really means that subsequent requests should hit > the same server. > > If you're using sessions, Tomcat has an attribute on the Engine > element called jvmRoute. So depending on your load balancer (and > if you use AJP), you can use Tomcat and AJP to route traffic. In > that case, there's no need to write a special cookie. > > At any rate, this doesn't sound like a clustering error per se. > >> >> ** What are the errors? Test result errors?
Re: Help with Tomcat 7 clustering using BIO receiver
Hello Filip The nodes are in the same machine! Regards João
Re: Help with Tomcat 7 clustering using BIO receiver
1. are your machines in time sync? If they are not, a session can get timed out. 2. 3. SEVERE: Manager [localhost#/myApp]: Unable to receive message through TCP channel 4. java.lang.IllegalStateException: setAttribute: Session [ DEC3612CF763194E7953DB3FD2C433E0] has already been invalidated 5. at org.apache.catalina.session.StandardSession.setAttribute( StandardSession.java:1437) 6. at org.apache.catalina.ha.session.DeltaSession.setAttribute( DeltaSession.java:695) 7. at org.apache.catalina.ha.session.DeltaRequest.execute( DeltaRequest.java:168) 8. at org.apache.catalina.ha.session.DeltaManager.handleSESSION_DELTA( DeltaManager.java:1337) 9. at org.apache.catalina.ha.session.DeltaManager.messageReceived (DeltaManager.java:1283) 10. at org.apache.catalina.ha.session.DeltaManager.messageDataReceived( DeltaManager.java:1001) 11. at org.apache.catalina.ha.session.ClusterSessionListener.messageReceived( ClusterSessionListener.java:91) 12. at org.apache.catalina.ha.tcp.SimpleTcpCluster.messageReceived( SimpleTcpCluster.java:943) 13. at org.apache.catalina.ha.tcp.SimpleTcpCluster.messageReceived( SimpleTcpCluster.java:924) 14. at org.apache.catalina.tribes.group.GroupChannel.messageReceived( GroupChannel.java:278) On Thu, Jul 3, 2014 at 1:07 PM, João Sávio wrote: > Hi everyone > > I ran my test (total of 1k requests, total of 100 threads) against two > nodes with default VM settings. I've just set heap size. I had about 15% of > errors. > > cluster.log - node1 - http://pastebin.com/cpX900Qw > cluster.log - node2 - http://pastebin.com/qCSzMaU6 > > Running for a long time (total of 500k requests, total of 100 threads) I > had about 11% of errors. In this case we can see the statistics: > > Jul 03, 2014 5:53:28 PM > org.apache.catalina.tribes.group.interceptors.ThroughputInterceptor report > INFO: ThroughputInterceptor Report[ > Tx Msg:1 messages > Sent:12.36 MB (total) > Sent:12.36 MB (application) > Time:7.82 seconds > Tx Speed:1.58 MB/sec (total) > TxSpeed:1.58 MB/sec (application) > Error Msg:0 > Rx Msg:10198 messages > Rx Speed:0.08 MB/sec (since 1st msg) > Received:12.38 MB] > > > All session attributes are Serializable, and it's a session replication > issue because if I ran my test with just one node, I had 0% of errors. > > Regarding "on time", just a correction: > > 1. first request, pick a random server and store a session object > 2. second request, pick *ANY* server (chose by LB based on the cookie - it > can be the same, but not necessarily) and ask for the session object. > > To be more clean, I've been working with a conference system. Each > conference should occur in one node. So, the first request can hit any > server, and from the second request should hit the node where the > conference is. > > > Thanks a lot > João > > > 2014-07-03 15:40 GMT-03:00 Filip Hanik : > > > you mention NIO and say maxThreads, that sounds like the > > configuration, but the BIO receiver is on the cluster, and it a > completely > > different component that also has an applicable NIO configuration. > > > > are you confusing the two? > > I'm saying that you should use the NIO receiver on the cluster component, > > and if you do, what kind of errors do you get? > > > > > > On Thu, Jul 3, 2014 at 12:19 PM, Mark Eggers > > > > > wrote: > > > > > -BEGIN PGP SIGNED MESSAGE- > > > Hash: SHA1 > > > > > > João, > > > > > > This list has a convention of posting either inline or at the end of > > > the message you're replying to. > > > > > > See here for mailing list notes: > > > > > > http://tomcat.apache.org/lists.html#tomcat-users > > > > > > On 7/3/2014 10:24 AM, João Sávio wrote: > > > > Hello > > > > > > > > Some points below: > > > > > > > > ** What is "on time"?* In my application, a group of users should > > > > always hit the same node after the first request. So, in first > > > > request each group of users will receive an specific cookie, and > > > > LB will perform the load balancing based on this cookie. In first > > > > request, a user can hit any node, but from the second, he or she > > > > should hit the same node. > > > > > > Hmm, so 'on time' really means that subsequent requests should hit the > > > same server. > > > > > > If you're using sessions, Tomcat has an attribute on the Engine > > > element called jvmRoute. So depending on your load balancer (and if > > > you use AJP), you can use Tomcat and AJP to route traffic. In that > > > case, there's no need to write a special cookie. > > > > > > At any rate, this doesn't sound like a clustering error per se. > > > > > > > > > > > ** What are the errors? Test result errors?* For this test, I > > > > simplified the code of my application: - first request: store one > > > > object in session - second request: verify if the object is in > > > > session. If it's not -
Re: Help with Tomcat 7 clustering using BIO receiver
Hi everyone I ran my test (total of 1k requests, total of 100 threads) against two nodes with default VM settings. I've just set heap size. I had about 15% of errors. cluster.log - node1 - http://pastebin.com/cpX900Qw cluster.log - node2 - http://pastebin.com/qCSzMaU6 Running for a long time (total of 500k requests, total of 100 threads) I had about 11% of errors. In this case we can see the statistics: Jul 03, 2014 5:53:28 PM org.apache.catalina.tribes.group.interceptors.ThroughputInterceptor report INFO: ThroughputInterceptor Report[ Tx Msg:1 messages Sent:12.36 MB (total) Sent:12.36 MB (application) Time:7.82 seconds Tx Speed:1.58 MB/sec (total) TxSpeed:1.58 MB/sec (application) Error Msg:0 Rx Msg:10198 messages Rx Speed:0.08 MB/sec (since 1st msg) Received:12.38 MB] All session attributes are Serializable, and it's a session replication issue because if I ran my test with just one node, I had 0% of errors. Regarding "on time", just a correction: 1. first request, pick a random server and store a session object 2. second request, pick *ANY* server (chose by LB based on the cookie - it can be the same, but not necessarily) and ask for the session object. To be more clean, I've been working with a conference system. Each conference should occur in one node. So, the first request can hit any server, and from the second request should hit the node where the conference is. Thanks a lot João 2014-07-03 15:40 GMT-03:00 Filip Hanik : > you mention NIO and say maxThreads, that sounds like the > configuration, but the BIO receiver is on the cluster, and it a completely > different component that also has an applicable NIO configuration. > > are you confusing the two? > I'm saying that you should use the NIO receiver on the cluster component, > and if you do, what kind of errors do you get? > > > On Thu, Jul 3, 2014 at 12:19 PM, Mark Eggers > > wrote: > > > -BEGIN PGP SIGNED MESSAGE- > > Hash: SHA1 > > > > João, > > > > This list has a convention of posting either inline or at the end of > > the message you're replying to. > > > > See here for mailing list notes: > > > > http://tomcat.apache.org/lists.html#tomcat-users > > > > On 7/3/2014 10:24 AM, João Sávio wrote: > > > Hello > > > > > > Some points below: > > > > > > ** What is "on time"?* In my application, a group of users should > > > always hit the same node after the first request. So, in first > > > request each group of users will receive an specific cookie, and > > > LB will perform the load balancing based on this cookie. In first > > > request, a user can hit any node, but from the second, he or she > > > should hit the same node. > > > > Hmm, so 'on time' really means that subsequent requests should hit the > > same server. > > > > If you're using sessions, Tomcat has an attribute on the Engine > > element called jvmRoute. So depending on your load balancer (and if > > you use AJP), you can use Tomcat and AJP to route traffic. In that > > case, there's no need to write a special cookie. > > > > At any rate, this doesn't sound like a clustering error per se. > > > > > > > > ** What are the errors? Test result errors?* For this test, I > > > simplified the code of my application: - first request: store one > > > object in session - second request: verify if the object is in > > > session. If it's not -> ERROR > > > > > > > So looking at the information from 'on time', the scenario should be: > > > > 1. first request, pick a random server and store a session object > > 2. second request, pick the SAME server and ask for the session object > > > > Again, I'm not seeing where this is a clustering issue per se. > > > > > ** How big are are the sessions that you're trying to replicate?* > > > - I'm using Spring MVC, and I have 3 additional objects in > > > session. They are not big (15 attributes each one) > > > > > > > And all attributes are serializable? The objects are also marked as > > serializable? > > > > > ** What's the load like on the box when you're running the tests > > > that you get errors on?* - I've experiencing this issue on BIO > > > even without load > > > > > > > I may have not phrased my question carefully. What is the CPU and > > memory situation on your test box while running the 4 Tomcat servers? > > > > I know you've trimmed down your Xms and Xmx (presumably to fit in your > > test environment), but in combination with your other JVM parameters > > could this be causing some issues? > > > > I would follow Dan's recommendation of maybe just setting Xms, Xmx, GC > > logging to see what happens. Ah, I see you're going to do that below. > > > > > ** It is preferred to use the non blocking receiver to be able to > > > grow your cluster without running into thread starvation.* - > > > That's why I've tried NIO first, but I'd like to see if BIO solve > > > my issue and if using BIO my system doesn't get too slow. > > > > I don't think speed is so much an issue here, but scalability is. NIO > > can handle mult
Re: Help with Tomcat 7 clustering using BIO receiver
you mention NIO and say maxThreads, that sounds like the configuration, but the BIO receiver is on the cluster, and it a completely different component that also has an applicable NIO configuration. are you confusing the two? I'm saying that you should use the NIO receiver on the cluster component, and if you do, what kind of errors do you get? On Thu, Jul 3, 2014 at 12:19 PM, Mark Eggers wrote: > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA1 > > João, > > This list has a convention of posting either inline or at the end of > the message you're replying to. > > See here for mailing list notes: > > http://tomcat.apache.org/lists.html#tomcat-users > > On 7/3/2014 10:24 AM, João Sávio wrote: > > Hello > > > > Some points below: > > > > ** What is "on time"?* In my application, a group of users should > > always hit the same node after the first request. So, in first > > request each group of users will receive an specific cookie, and > > LB will perform the load balancing based on this cookie. In first > > request, a user can hit any node, but from the second, he or she > > should hit the same node. > > Hmm, so 'on time' really means that subsequent requests should hit the > same server. > > If you're using sessions, Tomcat has an attribute on the Engine > element called jvmRoute. So depending on your load balancer (and if > you use AJP), you can use Tomcat and AJP to route traffic. In that > case, there's no need to write a special cookie. > > At any rate, this doesn't sound like a clustering error per se. > > > > > ** What are the errors? Test result errors?* For this test, I > > simplified the code of my application: - first request: store one > > object in session - second request: verify if the object is in > > session. If it's not -> ERROR > > > > So looking at the information from 'on time', the scenario should be: > > 1. first request, pick a random server and store a session object > 2. second request, pick the SAME server and ask for the session object > > Again, I'm not seeing where this is a clustering issue per se. > > > ** How big are are the sessions that you're trying to replicate?* > > - I'm using Spring MVC, and I have 3 additional objects in > > session. They are not big (15 attributes each one) > > > > And all attributes are serializable? The objects are also marked as > serializable? > > > ** What's the load like on the box when you're running the tests > > that you get errors on?* - I've experiencing this issue on BIO > > even without load > > > > I may have not phrased my question carefully. What is the CPU and > memory situation on your test box while running the 4 Tomcat servers? > > I know you've trimmed down your Xms and Xmx (presumably to fit in your > test environment), but in combination with your other JVM parameters > could this be causing some issues? > > I would follow Dan's recommendation of maybe just setting Xms, Xmx, GC > logging to see what happens. Ah, I see you're going to do that below. > > > ** It is preferred to use the non blocking receiver to be able to > > grow your cluster without running into thread starvation.* - > > That's why I've tried NIO first, but I'd like to see if BIO solve > > my issue and if using BIO my system doesn't get too slow. > > I don't think speed is so much an issue here, but scalability is. NIO > can handle multiple requests per thread, BIO cannot. > > > > > > > Now, I'll try to run my tests using NIO, default VM configuration > > and FINER logs. > > Post the results when you get them. If the logs are relatively small, > just cut and paste into the mail message. > > I suspect FINER is going to generate LOTS of logging and slow down > your application. > > > > > Thanks a lot João > > . . . . just my two cents > /mde/ > > > > > > > 2014-07-03 14:07 GMT-03:00 Mark Eggers > > : > > > > On 7/3/2014 9:12 AM, João Sávio wrote: > cluster.log -> http://pastebin.com/c98WhnmG > > > 2014-07-03 13:04 GMT-03:00 João Sávio : > > > Hello! > > > > Using NIO (with channelSendOptions="4", i.e., > > synchronous), with lightly load, my tests pass 100%. But, > > on heavy load, not all sessions are replicated on time, and > > I have about 20% of errors. If I increase maxThreads to > > 400, I have about 15% of errors. > > > > More information: * I am not performing parallel requests > > with same session * my cluster has 4 nodes (all in one > > machine - for test purpose only) * Java 7 64 bits, Tomcat > > 7.0.52, windows 7 64 bits * using default NIO > > configuration, but with maxThreads=400 * VM options: > > -Xms512M - on real environment this value is 1024 > > -Xmx512M - on real environment this value is 1024 > > -XX:NewSize=450M -XX:MaxNewSize=450M -XX:PermSize=128M > > -XX:MaxPermSize=245M -XX:SurvivorRatio=8 > > -XX:TargetSurvivorRatio=90 -XX:MaxTenuringThreshold=15 > > -XX:+UseBiasedLocking -XX:CMSInitiatingOccupancyFraction=60 > > -XX:+UseC
Re: Help with Tomcat 7 clustering using BIO receiver
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 João, This list has a convention of posting either inline or at the end of the message you're replying to. See here for mailing list notes: http://tomcat.apache.org/lists.html#tomcat-users On 7/3/2014 10:24 AM, João Sávio wrote: > Hello > > Some points below: > > ** What is "on time"?* In my application, a group of users should > always hit the same node after the first request. So, in first > request each group of users will receive an specific cookie, and > LB will perform the load balancing based on this cookie. In first > request, a user can hit any node, but from the second, he or she > should hit the same node. Hmm, so 'on time' really means that subsequent requests should hit the same server. If you're using sessions, Tomcat has an attribute on the Engine element called jvmRoute. So depending on your load balancer (and if you use AJP), you can use Tomcat and AJP to route traffic. In that case, there's no need to write a special cookie. At any rate, this doesn't sound like a clustering error per se. > > ** What are the errors? Test result errors?* For this test, I > simplified the code of my application: - first request: store one > object in session - second request: verify if the object is in > session. If it's not -> ERROR > So looking at the information from 'on time', the scenario should be: 1. first request, pick a random server and store a session object 2. second request, pick the SAME server and ask for the session object Again, I'm not seeing where this is a clustering issue per se. > ** How big are are the sessions that you're trying to replicate?* > - I'm using Spring MVC, and I have 3 additional objects in > session. They are not big (15 attributes each one) > And all attributes are serializable? The objects are also marked as serializable? > ** What's the load like on the box when you're running the tests > that you get errors on?* - I've experiencing this issue on BIO > even without load > I may have not phrased my question carefully. What is the CPU and memory situation on your test box while running the 4 Tomcat servers? I know you've trimmed down your Xms and Xmx (presumably to fit in your test environment), but in combination with your other JVM parameters could this be causing some issues? I would follow Dan's recommendation of maybe just setting Xms, Xmx, GC logging to see what happens. Ah, I see you're going to do that below. > ** It is preferred to use the non blocking receiver to be able to > grow your cluster without running into thread starvation.* - > That's why I've tried NIO first, but I'd like to see if BIO solve > my issue and if using BIO my system doesn't get too slow. I don't think speed is so much an issue here, but scalability is. NIO can handle multiple requests per thread, BIO cannot. > > > Now, I'll try to run my tests using NIO, default VM configuration > and FINER logs. Post the results when you get them. If the logs are relatively small, just cut and paste into the mail message. I suspect FINER is going to generate LOTS of logging and slow down your application. > > Thanks a lot João . . . . just my two cents /mde/ > > > 2014-07-03 14:07 GMT-03:00 Mark Eggers > : > > On 7/3/2014 9:12 AM, João Sávio wrote: cluster.log -> http://pastebin.com/c98WhnmG 2014-07-03 13:04 GMT-03:00 João Sávio : > Hello! > > Using NIO (with channelSendOptions="4", i.e., > synchronous), with lightly load, my tests pass 100%. But, > on heavy load, not all sessions are replicated on time, and > I have about 20% of errors. If I increase maxThreads to > 400, I have about 15% of errors. > > More information: * I am not performing parallel requests > with same session * my cluster has 4 nodes (all in one > machine - for test purpose only) * Java 7 64 bits, Tomcat > 7.0.52, windows 7 64 bits * using default NIO > configuration, but with maxThreads=400 * VM options: > -Xms512M - on real environment this value is 1024 > -Xmx512M - on real environment this value is 1024 > -XX:NewSize=450M -XX:MaxNewSize=450M -XX:PermSize=128M > -XX:MaxPermSize=245M -XX:SurvivorRatio=8 > -XX:TargetSurvivorRatio=90 -XX:MaxTenuringThreshold=15 > -XX:+UseBiasedLocking -XX:CMSInitiatingOccupancyFraction=60 > -XX:+UseCMSInitiatingOccupancyOnly > -XX:+CMSClassUnloadingEnabled -XX:+UseConcMarkSweepGC > -XX:+CMSIncrementalMode -XX:+UseParNewGC > -XX:+DisableExplicitGC -XX:+PrintGCDateStamps > -XX:+PrintGCDetails > -Xloggc:%CATALINA_BASE%/logs/tomcat-gc.log > > Moreover, I'm trying to attach the logs again. > > Thanks João > > João, > > I took a look at the log. This is the BIO attempt and you do run > out of threads. See the following: > > Jul 03, 2014 11:41:21 AM > org.apache.catalina.tribes.transport.bio.BioReceiver listen > WARNING: All BIO server replic
Re: Help with Tomcat 7 clustering using BIO receiver
Hello Some points below: ** What is "on time"?* In my application, a group of users should always hit the same node after the first request. So, in first request each group of users will receive an specific cookie, and LB will perform the load balancing based on this cookie. In first request, a user can hit any node, but from the second, he or she should hit the same node. ** What are the errors? Test result errors?* For this test, I simplified the code of my application: - first request: store one object in session - second request: verify if the object is in session. If it's not -> ERROR ** How big are are the sessions that you're trying to replicate?* - I'm using Spring MVC, and I have 3 additional objects in session. They are not big (15 attributes each one) ** What's the load like on the box when you're running the tests that you get errors on?* - I've experiencing this issue on BIO even without load ** It is preferred to use the non blocking receiver to be able to grow your cluster without running into thread starvation.* - That's why I've tried NIO first, but I'd like to see if BIO solve my issue and if using BIO my system doesn't get too slow. Now, I'll try to run my tests using NIO, default VM configuration and FINER logs. Thanks a lot João 2014-07-03 14:07 GMT-03:00 Mark Eggers : > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA1 > > On 7/3/2014 9:12 AM, João Sávio wrote: > > cluster.log -> http://pastebin.com/c98WhnmG > > > > > > 2014-07-03 13:04 GMT-03:00 João Sávio : > > > >> Hello! > >> > >> Using NIO (with channelSendOptions="4", i.e., synchronous), with > >> lightly load, my tests pass 100%. But, on heavy load, not all > >> sessions are replicated on time, and I have about 20% of errors. > >> If I increase maxThreads to 400, I have about 15% of errors. > >> > >> More information: * I am not performing parallel requests with > >> same session * my cluster has 4 nodes (all in one machine - for > >> test purpose only) * Java 7 64 bits, Tomcat 7.0.52, windows 7 64 > >> bits * using default NIO configuration, but with maxThreads=400 * > >> VM options: -Xms512M - on real environment this value is 1024 > >> -Xmx512M - on real environment this value is 1024 > >> -XX:NewSize=450M -XX:MaxNewSize=450M -XX:PermSize=128M > >> -XX:MaxPermSize=245M -XX:SurvivorRatio=8 > >> -XX:TargetSurvivorRatio=90 -XX:MaxTenuringThreshold=15 > >> -XX:+UseBiasedLocking -XX:CMSInitiatingOccupancyFraction=60 > >> -XX:+UseCMSInitiatingOccupancyOnly -XX:+CMSClassUnloadingEnabled > >> -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode -XX:+UseParNewGC > >> -XX:+DisableExplicitGC -XX:+PrintGCDateStamps > >> -XX:+PrintGCDetails -Xloggc:%CATALINA_BASE%/logs/tomcat-gc.log > >> > >> Moreover, I'm trying to attach the logs again. > >> > >> Thanks João > > João, > > I took a look at the log. This is the BIO attempt and you do run out > of threads. See the following: > > Jul 03, 2014 11:41:21 AM > org.apache.catalina.tribes.transport.bio.BioReceiver listen > WARNING: All BIO server replication threads are busy, unable to handle > more requests until a thread is freed up. > > What's the load like on the box when you're running the tests that you > get errors on? > > As Dan asks in his message: > > What is "on time"? > What are the errors? Test result errors? > > How big are are the sessions that you're trying to replicate? > > My guess is that something else is going on, since the following log > entry doesn't show much in the way of cluster traffic. > > INFO: ThroughputInterceptor Report[ > Tx Msg:1 messages > Sent:0.00 MB (total) > Sent:0.00 MB (application) > Time:0.01 seconds > Tx Speed:0.04 MB/sec (total) > TxSpeed:0.04 MB/sec (application) > Error Msg:0 > Rx Msg:13 messages > Rx Speed:0.00 MB/sec (since 1st msg) > Received:0.00 MB] > > It would also be interesting to see the logs when you use the NIO > connector. According to the documentation: > > It is preferred to use the non blocking receiver to be able to grow > your cluster without running into thread starvation. > > Also from the documentation: > > Usually the rule is to use 1 thread per node in the cluster for small > clusters, and then depending on your message frequency and your > hardware, you'll find an optimal number of threads peak out at a > certain number. > > We might need a little more background on your application and your > test environment to figure out why clustering is not behaving for you. > > . . . just my two cents > /mde/ > > PS - you have some errors in your server.xml (see the log). While they > won't impact this problem, it might be a good idea to address them. > > /mde/ > -BEGIN PGP SIGNATURE- > Version: GnuPG v1.4.13 (MingW32) > Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ > > iQEcBAEBAgAGBQJTtY28AAoJEEFGbsYNeTwtbk4H/1ehs00fmOLGfpcDxKkbfJJc > B2T3FEYmW2scV/W3Z0+z4uhBgVwRqPHgEZHotdRFhkadymCKz0d5RjjEgnTMv5vH > eP1u35NjmtteeLg+EcZU9
Re: Help with Tomcat 7 clustering using BIO receiver
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 7/3/2014 9:12 AM, João Sávio wrote: > cluster.log -> http://pastebin.com/c98WhnmG > > > 2014-07-03 13:04 GMT-03:00 João Sávio : > >> Hello! >> >> Using NIO (with channelSendOptions="4", i.e., synchronous), with >> lightly load, my tests pass 100%. But, on heavy load, not all >> sessions are replicated on time, and I have about 20% of errors. >> If I increase maxThreads to 400, I have about 15% of errors. >> >> More information: * I am not performing parallel requests with >> same session * my cluster has 4 nodes (all in one machine - for >> test purpose only) * Java 7 64 bits, Tomcat 7.0.52, windows 7 64 >> bits * using default NIO configuration, but with maxThreads=400 * >> VM options: -Xms512M - on real environment this value is 1024 >> -Xmx512M - on real environment this value is 1024 >> -XX:NewSize=450M -XX:MaxNewSize=450M -XX:PermSize=128M >> -XX:MaxPermSize=245M -XX:SurvivorRatio=8 >> -XX:TargetSurvivorRatio=90 -XX:MaxTenuringThreshold=15 >> -XX:+UseBiasedLocking -XX:CMSInitiatingOccupancyFraction=60 >> -XX:+UseCMSInitiatingOccupancyOnly -XX:+CMSClassUnloadingEnabled >> -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode -XX:+UseParNewGC >> -XX:+DisableExplicitGC -XX:+PrintGCDateStamps >> -XX:+PrintGCDetails -Xloggc:%CATALINA_BASE%/logs/tomcat-gc.log >> >> Moreover, I'm trying to attach the logs again. >> >> Thanks João João, I took a look at the log. This is the BIO attempt and you do run out of threads. See the following: Jul 03, 2014 11:41:21 AM org.apache.catalina.tribes.transport.bio.BioReceiver listen WARNING: All BIO server replication threads are busy, unable to handle more requests until a thread is freed up. What's the load like on the box when you're running the tests that you get errors on? As Dan asks in his message: What is "on time"? What are the errors? Test result errors? How big are are the sessions that you're trying to replicate? My guess is that something else is going on, since the following log entry doesn't show much in the way of cluster traffic. INFO: ThroughputInterceptor Report[ Tx Msg:1 messages Sent:0.00 MB (total) Sent:0.00 MB (application) Time:0.01 seconds Tx Speed:0.04 MB/sec (total) TxSpeed:0.04 MB/sec (application) Error Msg:0 Rx Msg:13 messages Rx Speed:0.00 MB/sec (since 1st msg) Received:0.00 MB] It would also be interesting to see the logs when you use the NIO connector. According to the documentation: It is preferred to use the non blocking receiver to be able to grow your cluster without running into thread starvation. Also from the documentation: Usually the rule is to use 1 thread per node in the cluster for small clusters, and then depending on your message frequency and your hardware, you'll find an optimal number of threads peak out at a certain number. We might need a little more background on your application and your test environment to figure out why clustering is not behaving for you. . . . just my two cents /mde/ PS - you have some errors in your server.xml (see the log). While they won't impact this problem, it might be a good idea to address them. /mde/ -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.13 (MingW32) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQEcBAEBAgAGBQJTtY28AAoJEEFGbsYNeTwtbk4H/1ehs00fmOLGfpcDxKkbfJJc B2T3FEYmW2scV/W3Z0+z4uhBgVwRqPHgEZHotdRFhkadymCKz0d5RjjEgnTMv5vH eP1u35NjmtteeLg+EcZU9XP1HOR+oxcx9fFic9NULtUb1lQOd9pIV9SWO82vFSI5 0ERzCxMr/ysiOZHPXPwl6SCe9TWGwYAWJh1QrH+3tqaD+EV7mYdZk7P/MOSWnSxn JzLRkO+nKPXLYv6NQiSzjCoyURIxv8+fIw3vIblx03vfhyKFlb/KR9r8ZfhlSiJ0 i9fKzpRXmCVIHchWCDWKV89l6KzOyIYPVv3LlprGyLtCTbaBqvBQu5iFOhbHHiw= =jgsE -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Help with Tomcat 7 clustering using BIO receiver
On Thu, Jul 3, 2014 at 12:04 PM, João Sávio wrote: > Hello! > > Using NIO (with channelSendOptions="4", i.e., synchronous), with lightly > load, my tests pass 100%. But, on heavy load, not all sessions are > replicated on time, > Define "on time". > and I have about 20% of errors. > Can you explain the errors more? Stack trace? > If I increase maxThreads to 400, I have about 15% of errors. > > More information: > * I am not performing parallel requests with same session > * my cluster has 4 nodes (all in one machine - for test purpose only) > * Java 7 64 bits, Tomcat 7.0.52, windows 7 64 bits > * using default NIO configuration, but with maxThreads=400 > * VM options: > -Xms512M - on real environment this value is 1024 > -Xmx512M - on real environment this value is 1024 > -XX:NewSize=450M > -XX:MaxNewSize=450M > -XX:PermSize=128M > -XX:MaxPermSize=245M > -XX:SurvivorRatio=8 > -XX:TargetSurvivorRatio=90 > -XX:MaxTenuringThreshold=15 > -XX:+UseBiasedLocking > -XX:CMSInitiatingOccupancyFraction=60 > -XX:+UseCMSInitiatingOccupancyOnly > -XX:+CMSClassUnloadingEnabled > -XX:+UseConcMarkSweepGC > -XX:+CMSIncrementalMode > -XX:+UseParNewGC > You have quite a few JVM settings configured here. No criticism to this, I assume that you've not copy and pasted these from the internet and have thoroughly tested each one. I'm just mentioning this because I often see people do this and it usually backfires. Just curious if you've tried things without all of these customizations. Perhaps with just some basics like -Xms, -Xmx and gc logging? > -XX:+DisableExplicitGC > -XX:+PrintGCDateStamps > -XX:+PrintGCDetails > -Xloggc:%CATALINA_BASE%/logs/tomcat-gc.log > > Moreover, I'm trying to attach the logs again. > The list doesn't like attachments. On some rare occasions it'll accept them, but it's generally better to inline the logs. Also some people, myself included, are paranoid and don't open attachments from people they do not know. Dan
Re: Help with Tomcat 7 clustering using BIO receiver
cluster.log -> http://pastebin.com/c98WhnmG 2014-07-03 13:04 GMT-03:00 João Sávio : > Hello! > > Using NIO (with channelSendOptions="4", i.e., synchronous), with lightly > load, my tests pass 100%. But, on heavy load, not all sessions are > replicated on time, and I have about 20% of errors. If I increase > maxThreads to 400, I have about 15% of errors. > > More information: > * I am not performing parallel requests with same session > * my cluster has 4 nodes (all in one machine - for test purpose only) > * Java 7 64 bits, Tomcat 7.0.52, windows 7 64 bits > * using default NIO configuration, but with maxThreads=400 > * VM options: > -Xms512M - on real environment this value is 1024 > -Xmx512M - on real environment this value is 1024 > -XX:NewSize=450M > -XX:MaxNewSize=450M > -XX:PermSize=128M > -XX:MaxPermSize=245M > -XX:SurvivorRatio=8 > -XX:TargetSurvivorRatio=90 > -XX:MaxTenuringThreshold=15 > -XX:+UseBiasedLocking > -XX:CMSInitiatingOccupancyFraction=60 > -XX:+UseCMSInitiatingOccupancyOnly > -XX:+CMSClassUnloadingEnabled > -XX:+UseConcMarkSweepGC > -XX:+CMSIncrementalMode > -XX:+UseParNewGC > -XX:+DisableExplicitGC > -XX:+PrintGCDateStamps > -XX:+PrintGCDetails > -Xloggc:%CATALINA_BASE%/logs/tomcat-gc.log > > Moreover, I'm trying to attach the logs again. > > Thanks > João > -- http://joaosavio.wordpress.com cluster.rar Description: application/rar - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Help with Tomcat 7 clustering using BIO receiver
Hello! Using NIO (with channelSendOptions="4", i.e., synchronous), with lightly load, my tests pass 100%. But, on heavy load, not all sessions are replicated on time, and I have about 20% of errors. If I increase maxThreads to 400, I have about 15% of errors. More information: * I am not performing parallel requests with same session * my cluster has 4 nodes (all in one machine - for test purpose only) * Java 7 64 bits, Tomcat 7.0.52, windows 7 64 bits * using default NIO configuration, but with maxThreads=400 * VM options: -Xms512M - on real environment this value is 1024 -Xmx512M - on real environment this value is 1024 -XX:NewSize=450M -XX:MaxNewSize=450M -XX:PermSize=128M -XX:MaxPermSize=245M -XX:SurvivorRatio=8 -XX:TargetSurvivorRatio=90 -XX:MaxTenuringThreshold=15 -XX:+UseBiasedLocking -XX:CMSInitiatingOccupancyFraction=60 -XX:+UseCMSInitiatingOccupancyOnly -XX:+CMSClassUnloadingEnabled -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode -XX:+UseParNewGC -XX:+DisableExplicitGC -XX:+PrintGCDateStamps -XX:+PrintGCDetails -Xloggc:%CATALINA_BASE%/logs/tomcat-gc.log Moreover, I'm trying to attach the logs again. Thanks João cluster.rar Description: application/rar - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Help with Tomcat 7 clustering using BIO receiver
I'd be more inclined to continue down the path of the NIO connector, it has been tested and used more. What are the errors you get when running with NIO? On Thu, Jul 3, 2014 at 8:51 AM, Konstantin Kolinko wrote: > 2014-07-03 18:46 GMT+04:00 João Sávio : > > Unfortunately it's not working yet > > > > I increased the log level as you suggested. Log attached > > > > Thanks > > > > Please read numbers 6. and 7. here: > http://tomcat.apache.org/lists.html#tomcat-users > > The attachment was thrown away by the mail server. > > - > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > >
Re: Help with Tomcat 7 clustering using BIO receiver
2014-07-03 18:46 GMT+04:00 João Sávio : > Unfortunately it's not working yet > > I increased the log level as you suggested. Log attached > > Thanks > Please read numbers 6. and 7. here: http://tomcat.apache.org/lists.html#tomcat-users The attachment was thrown away by the mail server. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Help with Tomcat 7 clustering using BIO receiver
Unfortunately it's not working yet I increased the log level as you suggested. Log attached Thanks 2014-07-03 11:10 GMT-03:00 João Sávio : > Hello Mark > > Thanks for your answer. I put the new information and cluster works (maybe > because I reboot my machine) > > Indeed, I'm trying BIO because NIO is causing session replication issues > on load tests. Using NIO, I've already tried to increase maxThreads, and > this minimizes the problem, but don't solve it. > > Thanks > > > 2014-07-02 13:28 GMT-03:00 Mark Eggers : > > -BEGIN PGP SIGNED MESSAGE- >> Hash: SHA1 >> >> On 7/2/2014 6:37 AM, João Sávio wrote: >> > ? >> > >> > >> > 2014-06-30 18:38 GMT-03:00 João Sávio : >> > >> >> Hello people >> >> >> >> This is my first message on this group! I'm trying to set up a >> >> Tomcat clustering using BIO receiver but I've been receiving the >> >> following error when I started two nodes and tried to enter on my >> >> application: >> >> >> >> Jun 30, 2014 9:25:12 PM >> >> org.apache.catalina.tribes.transport.bio.BioReplicationT ask run >> >> SEVERE: Unable to service bio socket >> >> java.net.SocketTimeoutException: Read timed out at >> >> java.net.SocketInputStream.socketRead0(Native Method) at >> >> java.net.SocketInputStream.read(SocketInputStream.java:152) at >> >> java.net.SocketInputStream.read(SocketInputStream.java:122) at >> >> java.net.SocketInputStream.read(SocketInputStream.java:108) at >> >> org.apache.catalina.tribes.transport.bio.BioReplicationTask.drainSock >> >> >> >> >> et(BioReplicationTask.java:146) >> >> at >> >> org.apache.catalina.tribes.transport.bio.BioReplicationTask.run(BioRe >> >> >> >> >> plicationTask.java:64) >> >> at >> >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor. >> >> >> >> >> java:1145) >> >> at >> >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor >> >> >> >> >> .java:615) >> >> at java.lang.Thread.run(Thread.java:744) >> >> >> >> Jun 30, 2014 9:25:15 PM >> >> org.apache.catalina.ha.session.DeltaManager startInternal INFO: >> >> Register manager localhost#/myApp to cluster element Engine with >> >> name Catalina Jun 30, 2014 9:25:15 PM >> >> org.apache.catalina.ha.session.DeltaManager startInternal INFO: >> >> Starting clustering manager at localhost#/myApp Jun 30, 2014 >> >> 9:25:15 PM org.apache.catalina.ha.session.DeltaManager >> >> getAllClusterSessions INFO: Manager [localhost#/myApp], >> >> requesting session state from >> >> org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 212, >> >> 191, 209}:4000,{10, 212, 191, 2 09},4000, alive=32096, >> >> securePort=-1, UDP Port=-1, id={-58 95 -116 89 86 -12 714 -75 -2 >> >> 74 -45 -54 -82 40 -114 }, payload={}, command={}, domain={}, ]. >> >> This operation will timeout if no session state has been received >> >> within 60 seconds. >> >> >> >> >> >> Attached are my two server.xml configuration. I've taken the >> >> default configuration (worked on my computer) and replaced the >> >> following things: >> >> >> >> >> >> > >> className="org.apache.catalina.tribes.transport.nio.NioReceiver" >> >> by > >> className="org.apache.catalina.tribes.transport.bio.BioReceiver" >> >> >> >> and >> >> >> >> > >> >> className="org.apache.catalina.tribes.transport.nio.PooledParallelSender"/> >> >> >> >> >> by >> >> > >> >> className="org.apache.catalina.tribes.transport.bio.PooledMultiSender"/> >> >> >> >> >> >> >> >> >> I don't know if I'm missing something. I'll be very thankful if someone >> >> can take a look on my server.xml files or send me an example >> >> configuration using BIO receiver (I couldn't find anyone). >> >> >> >> Thanks João -- http://joaosavio.wordpress.com >> >> João, >> >> Welcome to the list. >> >> I've only used NIO, and it's been a while since I've used clustering. >> >> A few questions to ask: >> >> 1. Is there a firewall blocking port 4000 (if the Tomcats are on >>separate machines)? >> 2. Multicast enabled and routed? >> 3. Environment (Tomcat version, java version, platforms)? >> 4. Why BIO? >> >> What are you trying to accomplish by using the BIO connector rather >> than the NIO connector? >> >> You can turn on more logging by adding something like the following to >> logging.properties: >> >> # at the end of the handlers line: >> ,5cluster.org.apache.juli.FileHandler >> >> # in the handler-specific properties section: >> 5cluster.org.apache.juli.FileHandler.level = FINER >> 5cluster.org.apache.juli.FileHandler.directory = ${catalina.base}/logs >> 5cluster.org.apache.juli.FileHandler.prefix = cluster. >> >> # in the facility specific properties section >> # be aware of line wrapping >> org.apache.catalina.tribes.MESSAGES.level = FINE >> org.apache.catalina.tribes.MESSAGES.handlers = >> 5cluster.org.apache.juli.FileHandler >> >> org.apache.catalina.tribes.level = FINE >> org.apache.catalina.tribes.handlers = 5cluster.org.apache.juli.FileHandler >> >> org.apache.catalina.ha.level = FINE >> org.apache.catalina.ha.handlers = 5cluster.org.apache.jul
Re: Help with Tomcat 7 clustering using BIO receiver
Hello Mark Thanks for your answer. I put the new information and cluster works (maybe because I reboot my machine) Indeed, I'm trying BIO because NIO is causing session replication issues on load tests. Using NIO, I've already tried to increase maxThreads, and this minimizes the problem, but don't solve it. Thanks 2014-07-02 13:28 GMT-03:00 Mark Eggers : > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA1 > > On 7/2/2014 6:37 AM, João Sávio wrote: > > ? > > > > > > 2014-06-30 18:38 GMT-03:00 João Sávio : > > > >> Hello people > >> > >> This is my first message on this group! I'm trying to set up a > >> Tomcat clustering using BIO receiver but I've been receiving the > >> following error when I started two nodes and tried to enter on my > >> application: > >> > >> Jun 30, 2014 9:25:12 PM > >> org.apache.catalina.tribes.transport.bio.BioReplicationT ask run > >> SEVERE: Unable to service bio socket > >> java.net.SocketTimeoutException: Read timed out at > >> java.net.SocketInputStream.socketRead0(Native Method) at > >> java.net.SocketInputStream.read(SocketInputStream.java:152) at > >> java.net.SocketInputStream.read(SocketInputStream.java:122) at > >> java.net.SocketInputStream.read(SocketInputStream.java:108) at > >> org.apache.catalina.tribes.transport.bio.BioReplicationTask.drainSock > >> > >> > et(BioReplicationTask.java:146) > >> at > >> org.apache.catalina.tribes.transport.bio.BioReplicationTask.run(BioRe > >> > >> > plicationTask.java:64) > >> at > >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor. > >> > >> > java:1145) > >> at > >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor > >> > >> > .java:615) > >> at java.lang.Thread.run(Thread.java:744) > >> > >> Jun 30, 2014 9:25:15 PM > >> org.apache.catalina.ha.session.DeltaManager startInternal INFO: > >> Register manager localhost#/myApp to cluster element Engine with > >> name Catalina Jun 30, 2014 9:25:15 PM > >> org.apache.catalina.ha.session.DeltaManager startInternal INFO: > >> Starting clustering manager at localhost#/myApp Jun 30, 2014 > >> 9:25:15 PM org.apache.catalina.ha.session.DeltaManager > >> getAllClusterSessions INFO: Manager [localhost#/myApp], > >> requesting session state from > >> org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 212, > >> 191, 209}:4000,{10, 212, 191, 2 09},4000, alive=32096, > >> securePort=-1, UDP Port=-1, id={-58 95 -116 89 86 -12 714 -75 -2 > >> 74 -45 -54 -82 40 -114 }, payload={}, command={}, domain={}, ]. > >> This operation will timeout if no session state has been received > >> within 60 seconds. > >> > >> > >> Attached are my two server.xml configuration. I've taken the > >> default configuration (worked on my computer) and replaced the > >> following things: > >> > >> > >> >> className="org.apache.catalina.tribes.transport.nio.NioReceiver" > >> by >> className="org.apache.catalina.tribes.transport.bio.BioReceiver" > >> > >> and > >> > >> >> > className="org.apache.catalina.tribes.transport.nio.PooledParallelSender"/> > >> > >> > by > >> >> className="org.apache.catalina.tribes.transport.bio.PooledMultiSender"/> > >> > >> > >> > >> > I don't know if I'm missing something. I'll be very thankful if someone > >> can take a look on my server.xml files or send me an example > >> configuration using BIO receiver (I couldn't find anyone). > >> > >> Thanks João -- http://joaosavio.wordpress.com > > João, > > Welcome to the list. > > I've only used NIO, and it's been a while since I've used clustering. > > A few questions to ask: > > 1. Is there a firewall blocking port 4000 (if the Tomcats are on >separate machines)? > 2. Multicast enabled and routed? > 3. Environment (Tomcat version, java version, platforms)? > 4. Why BIO? > > What are you trying to accomplish by using the BIO connector rather > than the NIO connector? > > You can turn on more logging by adding something like the following to > logging.properties: > > # at the end of the handlers line: > ,5cluster.org.apache.juli.FileHandler > > # in the handler-specific properties section: > 5cluster.org.apache.juli.FileHandler.level = FINER > 5cluster.org.apache.juli.FileHandler.directory = ${catalina.base}/logs > 5cluster.org.apache.juli.FileHandler.prefix = cluster. > > # in the facility specific properties section > # be aware of line wrapping > org.apache.catalina.tribes.MESSAGES.level = FINE > org.apache.catalina.tribes.MESSAGES.handlers = > 5cluster.org.apache.juli.FileHandler > > org.apache.catalina.tribes.level = FINE > org.apache.catalina.tribes.handlers = 5cluster.org.apache.juli.FileHandler > > org.apache.catalina.ha.level = FINE > org.apache.catalina.ha.handlers = 5cluster.org.apache.juli.FileHander > > org.apache.catalina.ha.deploy.level = INFO > org.apache.catalina.ha.deploy.handlers = > 5cluster.org.apache.juli.FileHandler > > Last note - we're normally a pretty helpful bunch of folks here, but > we are all volunteers. Things like work impact participation :-p. Also > as noted a
Re: Help with Tomcat 7 clustering using BIO receiver
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 7/2/2014 6:37 AM, João Sávio wrote: > ? > > > 2014-06-30 18:38 GMT-03:00 João Sávio : > >> Hello people >> >> This is my first message on this group! I'm trying to set up a >> Tomcat clustering using BIO receiver but I've been receiving the >> following error when I started two nodes and tried to enter on my >> application: >> >> Jun 30, 2014 9:25:12 PM >> org.apache.catalina.tribes.transport.bio.BioReplicationT ask run >> SEVERE: Unable to service bio socket >> java.net.SocketTimeoutException: Read timed out at >> java.net.SocketInputStream.socketRead0(Native Method) at >> java.net.SocketInputStream.read(SocketInputStream.java:152) at >> java.net.SocketInputStream.read(SocketInputStream.java:122) at >> java.net.SocketInputStream.read(SocketInputStream.java:108) at >> org.apache.catalina.tribes.transport.bio.BioReplicationTask.drainSock >> >> et(BioReplicationTask.java:146) >> at >> org.apache.catalina.tribes.transport.bio.BioReplicationTask.run(BioRe >> >> plicationTask.java:64) >> at >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor. >> >> java:1145) >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor >> >> .java:615) >> at java.lang.Thread.run(Thread.java:744) >> >> Jun 30, 2014 9:25:15 PM >> org.apache.catalina.ha.session.DeltaManager startInternal INFO: >> Register manager localhost#/myApp to cluster element Engine with >> name Catalina Jun 30, 2014 9:25:15 PM >> org.apache.catalina.ha.session.DeltaManager startInternal INFO: >> Starting clustering manager at localhost#/myApp Jun 30, 2014 >> 9:25:15 PM org.apache.catalina.ha.session.DeltaManager >> getAllClusterSessions INFO: Manager [localhost#/myApp], >> requesting session state from >> org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 212, >> 191, 209}:4000,{10, 212, 191, 2 09},4000, alive=32096, >> securePort=-1, UDP Port=-1, id={-58 95 -116 89 86 -12 714 -75 -2 >> 74 -45 -54 -82 40 -114 }, payload={}, command={}, domain={}, ]. >> This operation will timeout if no session state has been received >> within 60 seconds. >> >> >> Attached are my two server.xml configuration. I've taken the >> default configuration (worked on my computer) and replaced the >> following things: >> >> >> > className="org.apache.catalina.tribes.transport.nio.NioReceiver" >> by > className="org.apache.catalina.tribes.transport.bio.BioReceiver" >> >> and >> >> > className="org.apache.catalina.tribes.transport.nio.PooledParallelSender"/> >> >> by >> > className="org.apache.catalina.tribes.transport.bio.PooledMultiSender"/> >> >> >> >> I don't know if I'm missing something. I'll be very thankful if someone >> can take a look on my server.xml files or send me an example >> configuration using BIO receiver (I couldn't find anyone). >> >> Thanks João -- http://joaosavio.wordpress.com João, Welcome to the list. I've only used NIO, and it's been a while since I've used clustering. A few questions to ask: 1. Is there a firewall blocking port 4000 (if the Tomcats are on separate machines)? 2. Multicast enabled and routed? 3. Environment (Tomcat version, java version, platforms)? 4. Why BIO? What are you trying to accomplish by using the BIO connector rather than the NIO connector? You can turn on more logging by adding something like the following to logging.properties: # at the end of the handlers line: ,5cluster.org.apache.juli.FileHandler # in the handler-specific properties section: 5cluster.org.apache.juli.FileHandler.level = FINER 5cluster.org.apache.juli.FileHandler.directory = ${catalina.base}/logs 5cluster.org.apache.juli.FileHandler.prefix = cluster. # in the facility specific properties section # be aware of line wrapping org.apache.catalina.tribes.MESSAGES.level = FINE org.apache.catalina.tribes.MESSAGES.handlers = 5cluster.org.apache.juli.FileHandler org.apache.catalina.tribes.level = FINE org.apache.catalina.tribes.handlers = 5cluster.org.apache.juli.FileHandler org.apache.catalina.ha.level = FINE org.apache.catalina.ha.handlers = 5cluster.org.apache.juli.FileHander org.apache.catalina.ha.deploy.level = INFO org.apache.catalina.ha.deploy.handlers = 5cluster.org.apache.juli.FileHandler Last note - we're normally a pretty helpful bunch of folks here, but we are all volunteers. Things like work impact participation :-p. Also as noted above, I've only used NIO so I don't know why this would not work with BIO. . . . just my two cents /mde/ -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.13 (MingW32) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQEcBAEBAgAGBQJTtDM0AAoJEEFGbsYNeTwtkR8H/3JM2LPJBolSWp6kA2K+HJYK mOn5LahCs6cQcZEKs6JKZMS5aVz9A8CPc4/e/LDfGJ86SH1kB1KAwPOwNtFjijl/ hpQvQYe8WTIc8Mh6QtbA1jfJD67FBqDxl356+QSrmZJnuJTEz8zAy/6L1/Fvo08c HLYOjfkPj463TnDXJMUNIk1DSuHIdEnaxdXpM55ADTrkrJC06pM/wuU9znfpP/58 xKbkX05ODgdBLvP7sfs4vI4DoVWO81+w/4W3o7DC4y9UmqQhtTtFvSacGh8IimW2 G/+PVCbYUigKaeegqIVo9eCXeHLO7OBX6njW
Re: Help with Tomcat 7 clustering using BIO receiver
? 2014-06-30 18:38 GMT-03:00 João Sávio : > Hello people > > This is my first message on this group! I'm trying to set up a Tomcat > clustering using BIO receiver but I've been receiving the following error > when I started two nodes and tried to enter on my application: > > Jun 30, 2014 9:25:12 PM > org.apache.catalina.tribes.transport.bio.BioReplicationT > ask run > SEVERE: Unable to service bio socket > java.net.SocketTimeoutException: Read timed out > at java.net.SocketInputStream.socketRead0(Native Method) > at java.net.SocketInputStream.read(SocketInputStream.java:152) > at java.net.SocketInputStream.read(SocketInputStream.java:122) > at java.net.SocketInputStream.read(SocketInputStream.java:108) > at > org.apache.catalina.tribes.transport.bio.BioReplicationTask.drainSock > et(BioReplicationTask.java:146) > at > org.apache.catalina.tribes.transport.bio.BioReplicationTask.run(BioRe > plicationTask.java:64) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor. > java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor > .java:615) > at java.lang.Thread.run(Thread.java:744) > > Jun 30, 2014 9:25:15 PM org.apache.catalina.ha.session.DeltaManager > startInternal > INFO: Register manager localhost#/myApp to cluster element Engine with > name Catalina > Jun 30, 2014 9:25:15 PM org.apache.catalina.ha.session.DeltaManager > startInternal > INFO: Starting clustering manager at localhost#/myApp > Jun 30, 2014 9:25:15 PM org.apache.catalina.ha.session.DeltaManager > getAllClusterSessions > INFO: Manager [localhost#/myApp], requesting session state from > org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 212, 191, > 209}:4000,{10, 212, 191, 2 > 09},4000, alive=32096, securePort=-1, UDP Port=-1, id={-58 95 -116 89 86 > -12 714 -75 -2 74 -45 -54 -82 40 -114 }, payload={}, command={}, > domain={}, ]. This operation will timeout if no session state has been > received within 60 seconds. > > > Attached are my two server.xml configuration. I've taken the default > configuration (worked on my computer) and replaced the following things: > > > by > > and > > className="org.apache.catalina.tribes.transport.nio.PooledParallelSender"/> > by > className="org.apache.catalina.tribes.transport.bio.PooledMultiSender"/> > > > I don't know if I'm missing something. I'll be very thankful if someone > can take a look on my server.xml files or send me an example configuration > using BIO receiver (I couldn't find anyone). > > Thanks > João > -- > http://joaosavio.wordpress.com > > > -- http://joaosavio.wordpress.com
Help with Tomcat 7 clustering using BIO receiver
Hello people This is my first message on this group! I'm trying to set up a Tomcat clustering using BIO receiver but I've been receiving the following error when I started two nodes and tried to enter on my application: Jun 30, 2014 9:25:12 PM org.apache.catalina.tribes.transport.bio.BioReplicationT ask run SEVERE: Unable to service bio socket java.net.SocketTimeoutException: Read timed out at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.read(SocketInputStream.java:152) at java.net.SocketInputStream.read(SocketInputStream.java:122) at java.net.SocketInputStream.read(SocketInputStream.java:108) at org.apache.catalina.tribes.transport.bio.BioReplicationTask.drainSock et(BioReplicationTask.java:146) at org.apache.catalina.tribes.transport.bio.BioReplicationTask.run(BioRe plicationTask.java:64) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor. java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor .java:615) at java.lang.Thread.run(Thread.java:744) Jun 30, 2014 9:25:15 PM org.apache.catalina.ha.session.DeltaManager startInternal INFO: Register manager localhost#/myApp to cluster element Engine with name Catalina Jun 30, 2014 9:25:15 PM org.apache.catalina.ha.session.DeltaManager startInternal INFO: Starting clustering manager at localhost#/myApp Jun 30, 2014 9:25:15 PM org.apache.catalina.ha.session.DeltaManager getAllClusterSessions INFO: Manager [localhost#/myApp], requesting session state from org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 212, 191, 209}:4000,{10, 212, 191, 2 09},4000, alive=32096, securePort=-1, UDP Port=-1, id={-58 95 -116 89 86 -12 714 -75 -2 74 -45 -54 -82 40 -114 }, payload={}, command={}, domain={}, ]. This operation will timeout if no session state has been received within 60 seconds. Attached are my two server.xml configuration. I've taken the default configuration (worked on my computer) and replaced the following things: by I don't know if I'm missing something. I'll be very thankful if someone can take a look on my server.xml files or send me an example configuration using BIO receiver (I couldn't find anyone). Thanks João -- http://joaosavio.wordpress.com - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org