Re: Tomcat Clustering Support
On 11/10/18 10:12, Mark Thomas wrote: > If folks think this looks reasonable, I can create a BZ enhancement > request to implement it. https://bz.apache.org/bugzilla/show_bug.cgi?id=62841 - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat Clustering Support
On 10/10/18 23:04, Caldarale, Charles R wrote: >> From: Mark Thomas [mailto:ma...@apache.org] >> Subject: Re: Tomcat Clustering Support > >> Thread A is in the middle of processing a request. It is evaluating some >> EL which requires access to the view map which in turn causes the >> ViewMap to update the session. >> com.sun.faces.application.view.ViewScopeManager.processEvent locks the >> ViewMap object. It then tries to update the session. To do this it >> requires the session lock. Thread A is waiting for this lock. > > Assuming the ViewMap is used by multiple sessions, this locking order goes > against the usual protocol of more local before more global. Might be > possible to file a bug report with Mojarra, but given that the code appears > to be in a com.sun class, that might not get anywhere. > >> Thread B is at the end of a request. The session has been updated and it >> is attempting to write the updated session attributes to the cluster. >> The session lock has been obtained. The individual attributes are being >> written. The code has reached the ViewMap object. In order to write this >> object, the ViewMap object must be locked. Thread B is waiting for this >> lock. > > This is the generally the more desirable order. I think ViewMap is per session but I haven't looked that closely at the code. >> Has anyone on the users list come across this problem before? If so, how >> have you solved it? Suggestions for alternative solutions also welcome. > > Can the thread doing the session synchronization lock the session, get a > shallow copy of the attributes, unlock the session, then process the > attributes? Not sure if that would maintain sufficient coherency. A variation of that might work but at the possible expense of generating rather more garbage. The changes to the session are stored in a DeltaRequest. Currently the sequence is: - lock session - serialize DeltaRequest to message - recycle DeltaRequest - unlock session - send message Change that to: - lock session - keep reference to populated DeltaRequest - provide session with new DeltaRequest object - unlock session - serialize populated DeltaRequest to message - send message and this deadlock should be resolved. To avoid the expense of creating a new DeltaRequest each time, a pool of them could be used which should minimise the garbage. Looking at the sequence of events, I don't think this does much that is likely to harm coherence. If folks think this looks reasonable, I can create a BZ enhancement request to implement it. Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Tomcat Clustering Support
> From: Mark Thomas [mailto:ma...@apache.org] > Subject: Re: Tomcat Clustering Support > Thread A is in the middle of processing a request. It is evaluating some > EL which requires access to the view map which in turn causes the > ViewMap to update the session. > com.sun.faces.application.view.ViewScopeManager.processEvent locks the > ViewMap object. It then tries to update the session. To do this it > requires the session lock. Thread A is waiting for this lock. Assuming the ViewMap is used by multiple sessions, this locking order goes against the usual protocol of more local before more global. Might be possible to file a bug report with Mojarra, but given that the code appears to be in a com.sun class, that might not get anywhere. > Thread B is at the end of a request. The session has been updated and it > is attempting to write the updated session attributes to the cluster. > The session lock has been obtained. The individual attributes are being > written. The code has reached the ViewMap object. In order to write this > object, the ViewMap object must be locked. Thread B is waiting for this > lock. This is the generally the more desirable order. > Has anyone on the users list come across this problem before? If so, how > have you solved it? Suggestions for alternative solutions also welcome. Can the thread doing the session synchronization lock the session, get a shallow copy of the attributes, unlock the session, then process the attributes? Not sure if that would maintain sufficient coherency. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. smime.p7s Description: S/MIME cryptographic signature
Re: Tomcat Clustering Support
On 15/08/18 20:52, Mark Thomas wrote: > On 15/08/18 20:43, Scott Evans wrote: >> Hi, >> >> Our system is on Apache Tomcat Version 8.0.47. >> OS is Windows Server 2012 R2 Datacenter. >> >> We are looking for someone that may be interested in paid contract work to >> assist with troubleshooting and resolving a Tomcat clustering issue in our >> system. >> >> The system is composed of multiple Java PrimeFaces applications running in >> a clustered Tomcat environment which is experiencing occasional >> deadlocking issues from an unknown source requiring the Nodes to be cycled >> in order to resolve. The issue is only occurring in our Production >> environment and we've determined that the issues are occurring at random >> with the replication threads. >> >> We would need someone to help investigate our configuration and determine >> if there are any further changes that can be made to our system to catch >> these deadlock issues before they occur (requiring a Node cycle). >> >> Please let me know if you or someone you know may be interested or if you >> have further questions I can help answer. > > If you can provide a thread dump of the deadlock when it occurs we can > probably help you here for free. Scott provided me with a sanitised copy of the thread-dump off-line. I'm sharing my analysis with the list (with Scott's permission) as I think the root cause is likely to be of wider interest. There was, indeed, a deadlock. The issues was follows. The application is using JSF. Specifically, the Mojarra implementation from Oracle. There are multiple concurrent requests for the same session. Each request is processed by a dedicated thread (this is mandated by the Servlet spec although it may not be expressed that way). The threads in question are: A. ajp-apr-8009-exec-9005 B. ajp-apr-8009-exec-9000 Thread A is in the middle of processing a request. It is evaluating some EL which requires access to the view map which in turn causes the ViewMap to update the session. com.sun.faces.application.view.ViewScopeManager.processEvent locks the ViewMap object. It then tries to update the session. To do this it requires the session lock. Thread A is waiting for this lock. Thread B is at the end of a request. The session has been updated and it is attempting to write the updated session attributes to the cluster. The session lock has been obtained. The individual attributes are being written. The code has reached the ViewMap object. In order to write this object, the ViewMap object must be locked. Thread B is waiting for this lock. So, thread A holds the lock that thread B wants and is waiting for the lock thread B is holding. Thread B holds the lock the thread A wants and is waiting for the lock thread A is holding. Deadlock. This is, in essence, cause by a combination of how Tomcat's clustering is designed and Mojarra is implemented. The application is using the BackupManager. I assume with sticky sessions. Therefore, I would expect session failover between nodes to be a rare event. My recommendation is to investigate excluding the ViewMap from the replication via sessionAttributeNameFilter. You'd need a regular expression that matched anything except "com.sun.faces.application.view.activeViewMaps" I don't know how integral this object is to Mojarra. Mojarra may simply recreate this object if required. If not, you may need to trigger recreation after failover. I don't know how feasible this solution is. This will require some testing and possibly code changes. Has anyone on the users list come across this problem before? If so, how have you solved it? Suggestions for alternative solutions also welcome. Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat Clustering Support
On 15/08/18 20:43, Scott Evans wrote: > Hi, > > Our system is on Apache Tomcat Version 8.0.47. > OS is Windows Server 2012 R2 Datacenter. > > We are looking for someone that may be interested in paid contract work to > assist with troubleshooting and resolving a Tomcat clustering issue in our > system. > > The system is composed of multiple Java PrimeFaces applications running in > a clustered Tomcat environment which is experiencing occasional > deadlocking issues from an unknown source requiring the Nodes to be cycled > in order to resolve. The issue is only occurring in our Production > environment and we've determined that the issues are occurring at random > with the replication threads. > > We would need someone to help investigate our configuration and determine > if there are any further changes that can be made to our system to catch > these deadlock issues before they occur (requiring a Node cycle). > > Please let me know if you or someone you know may be interested or if you > have further questions I can help answer. If you can provide a thread dump of the deadlock when it occurs we can probably help you here for free. Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat Clustering Support - Without any load balancer like component in the front
Thanks for the quick reply Gary.. Some follow-up questions. 1) So the http server bundled with tomcat doesn't do any help ? like it doesnt provide any load-balancing implementation ? like e.g. mod-jk which has to be used in conjunction with apache...Further I presume mod-jk also handles the fail-over scenarios.. 2) When u say there is no huge amount of point, i wonder if there is any point AT ALL having that tomcat clustering enabled if we dont have something like a load-balancer/fail-over-service at the front? Thanks again Farhan. Gary Evesson-3 wrote: You need something to handle failover. Otherwise there is not a huge amount of point. Either a load balancer or mod-jk will do the job. On Nov 27, 2007 4:13 PM, mfs [EMAIL PROTECTED] wrote: Guys, Pretty basic question, given this is my time experience on clustering where i am trying to use tomcat 6 clustering support. So basically i wanted to know if enabling the tomcat 6 clustering would be of any use without having a load-balancer in the front (something like mod_jk) ? well my understanding so far is that we have to have some component in the front (which tomcat i assume doesn't provide...right ?) for either load-balancing or fail-over scenarios... Thanks in advance and REgards, Farhan. -- View this message in context: http://www.nabble.com/Tomcat-Clustering-Support---Without-any-load-balancer-like-component-in-the-front-tf4885156.html#a13982315 Sent from the Tomcat - User mailing list archive at Nabble.com. - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] -- Gary Evesson -- View this message in context: http://www.nabble.com/Tomcat-Clustering-Support---Without-any-load-balancer-like-component-in-the-front-tf4885156.html#a13982961 Sent from the Tomcat - User mailing list archive at Nabble.com. - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Tomcat Clustering Support - Without any load balancer like component in the front
mod_jk handles failover for you. Clustering tomcat makes it copy sessions to all members of the clusters so that any member of the cluster can take over the processing of a request seamlessly from any other. If you requests have no way of getting to the other members of the cluster, then there is not much point in clustering. You can use something like pen or a dedicated load balancer to handle failover (and load balancing if you like) if you do not want to set up apache. On Nov 27, 2007 5:01 PM, mfs [EMAIL PROTECTED] wrote: Thanks for the quick reply Gary.. Some follow-up questions. 1) So the http server bundled with tomcat doesn't do any help ? like it doesnt provide any load-balancing implementation ? like e.g. mod-jk which has to be used in conjunction with apache...Further I presume mod-jk also handles the fail-over scenarios.. 2) When u say there is no huge amount of point, i wonder if there is any point AT ALL having that tomcat clustering enabled if we dont have something like a load-balancer/fail-over-service at the front? Thanks again Farhan. Gary Evesson-3 wrote: You need something to handle failover. Otherwise there is not a huge amount of point. Either a load balancer or mod-jk will do the job. On Nov 27, 2007 4:13 PM, mfs [EMAIL PROTECTED] wrote: Guys, Pretty basic question, given this is my time experience on clustering where i am trying to use tomcat 6 clustering support. So basically i wanted to know if enabling the tomcat 6 clustering would be of any use without having a load-balancer in the front (something like mod_jk) ? well my understanding so far is that we have to have some component in the front (which tomcat i assume doesn't provide...right ?) for either load-balancing or fail-over scenarios... Thanks in advance and REgards, Farhan. -- View this message in context: http://www.nabble.com/Tomcat-Clustering-Support---Without-any-load-balancer-like-component-in-the-front-tf4885156.html#a13982315 Sent from the Tomcat - User mailing list archive at Nabble.com. - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] -- Gary Evesson -- View this message in context: http://www.nabble.com/Tomcat-Clustering-Support---Without-any-load-balancer-like-component-in-the-front-tf4885156.html#a13982961 Sent from the Tomcat - User mailing list archive at Nabble.com. - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] -- Gary Evesson
