Finally, I switched back to the asynchronous replication, just set the replication interval to 500 ms and I hope it will be fast enough for the OAuth application. The load tests did not show any surprising change, so I will keep it like this.
I’m still not sure what was the root cause of the problem; maybe a random database issue (we check users against a database) or some momentary network issues that caused the synchronous replication to take that long. Anyway, I found that the login request holds a Spring Webflow Execution ID, so a repeated submission of the same login request can really cause the locking problem in question. Thank you once more for your help! Best Regards, Jarda -------------------------------------------------- Jaroslav KAČER IDC | Web Channel Team | Application Developer E-Mail: [email protected]<mailto:[email protected]> Skype: jkacer.idc From: [email protected] [mailto:[email protected]] On Behalf Of Jaroslav Kacer Sent: 20. April 2016 10:05 dop. To: [email protected] Subject: RE: [cas-user] Synchronous replication of TGT - Is it safe? Ray, Raymond, Misagh, Thank you very much for your answers. For the moment, I reverted the configuration back to asynchronous replication but decreased the period between two consecutive replications dramatically – from 10 seconds to 500 ms. This should help us to solve the original issue with TGTs not available on the other node, although I don’t think it’s a real solution. I’m little afraid of a potential performance impact, so I am going to load-test it now on a cluster of 2 nodes. Unfortunately, I cannot do what Ray did, we need all nodes of the cluster due to busy traffic. We have 4 nodes in production. Yesterday, I was experimenting a little bit and introduced an artificial delay of 50 seconds to the central authentication service, to the point where authenticate() is called. Then I tried to log in from multiple browsers simultaneously. Surprisingly, I did not get the locking exception. I got another exception on unlock() caused by an expired session but the lock was properly unlocked. It seems the kind of exceptions we normally experience only occurs if the same user somehow reposts the login request from the same browser. Then the locking exception would make sense to me but I have to verify this hypothesis. I’ll post my findings here. Thank you once again for your help, gentlemen. Jarda -------------------------------------------------- Jaroslav KAČER IDC | Web Channel Team | Application Developer E-Mail: [email protected]<mailto:[email protected]> Skype: jkacer.idc From: [email protected]<mailto:[email protected]> [mailto:[email protected]] On Behalf Of Misagh Moayyed Sent: 19. April 2016 11:18 odp. To: [email protected]<mailto:[email protected]> Subject: RE: [cas-user] Synchronous replication of TGT - Is it safe? I know of folks who are doing this sort of thing asynchronously with success. It very much depends on your network topology, bandwidth and latency, whether nodes are on the WAN and your expected load. We can certainly get the docs changed or at a minimum, describe the case and option available for those who’d want it. From: [email protected]<mailto:[email protected]> [mailto:[email protected]] On Behalf Of Raymond Drew Walker Sent: Tuesday, April 19, 2016 10:31 AM To: Jaroslav Kacer <[email protected]<mailto:[email protected]>>; [email protected]<mailto:[email protected]> Subject: Re: [cas-user] Synchronous replication of TGT - Is it safe? I’ve brought this topic up on this list multiple times with no answer, most recently: https://groups.google.com/d/msg/jasig-cas-user/YOEFTnseBjs/msFyW0P-fk4J (our environment info is in this thread for reference) We’ve run into similar issues, as we’re concerned with immediate (sub-second) TGT availability. Even with some java/memory tuning tweaks, the issue was uncorrectable. Alas our solution to date was switching to synchronous TGT replication which has served us well for the past year or so now with no issue. You may have other issues, as I don’t recall anything similar to your Spring locking issues. Admins, Could we get the docs changed if no one has a reason for TGT replication via EHCache to be asynchronous? — Raymond Walker Software Systems Engineer StSp. ITS Northern Arizona University From: <[email protected]<mailto:[email protected]>> on behalf of Jaroslav Kacer <[email protected]<mailto:[email protected]>> Date: Tuesday, April 19, 2016 at 4:51 AM To: "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Subject: [cas-user] Synchronous replication of TGT - Is it safe? Hello everyone! We use CAS 4.0 with EH Cache ticket registry in a clustered environment. Initially we set up the registry and ticket replication according to this documentation: http://jasig.github.io/cas/4.0.x/installation/Ehcache-Ticket-Registry.html It uses synchronous replication for STs and asynchronous replication of TGTs. This configuration worked just fine for us, until we deployed an application that uses CAS as OAuth token provider. We immediately hit a problem that the TGT was not always available on the 2nd node of CAS cluster before the application wanted to verify the OAuth token (=TGT); it was a question of 1-2 seconds, sometimes it was already available, sometimes not. Therefore, we switched over to synchronous replication of TGTs, which solved this problem. However, I am not 100% sure this is a safe solution. May I ask if someone has already used synchronous replication for TGTs? I’m asking because since then, we have been randomly getting long delays when validation user credentials on login. It can be seen in the logs that these delays are cause by a log inside Spring Webflow: SEVERE: Servlet.service() for servlet [cas] in context with path [] threw exception [Request processing failed; nested exception is org.springframework.webflow.conversation.impl.LockTimeoutException: Unable to acquire conversation lock after 30 seconds] with root cause org.springframework.webflow.conversation.impl.LockTimeoutException: Unable to acquire conversation lock after 30 seconds at org.springframework.webflow.conversation.impl.JdkConcurrentConversationLock.lock(JdkConcurrentConversationLock.java:44) at org.springframework.webflow.conversation.impl.ContainedConversation.lock(ContainedConversation.java:69) at org.springframework.webflow.execution.repository.support.ConversationBackedFlowExecutionLock.lock(ConversationBackedFlowExecutionLock.java:51) at org.springframework.webflow.executor.FlowExecutorImpl.resumeExecution(FlowExecutorImpl.java:166) at org.springframework.webflow.mvc.servlet.FlowHandlerAdapter.handle(FlowHandlerAdapter.java:183) at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:925) This happens randomly, about once per day. I found some related information in CAS Jira here: https://issues.jasig.org/browse/CAS-993 And also some older discussion on this list: https://groups.google.com/forum/#!topic/jasig-cas-user/k3wcrKpO69c The Jira ticket says an authentication handler that takes too long can cause such errors. But I’m not sure I understand it completely. Specifically, I’d like to know: · Can synchronous replication of a single TGT cause the authentication handler not to return at all or return after a long time? I would say it can cause long delays only but I’m not sure. · Even if the authentication handler is still running, how can we get the timeout on the lock when all this happens after a single form post, i.e. in a single request, i.e. in a single thread? The lock belongs to a particular web flow execution and just the thread processing the post request should try to lock it, shouldn’t it? I would really appreciate if someone could share his experience here or explain me what exactly happens there when the lock cannot be acquired. Thank you! Best Regards, Jarda -------------------------------------------------- Jaroslav KAČER IDC | Web Channel Team | Application Developer E-Mail: [email protected]<mailto:[email protected]> Skype: jkacer.idc -- You received this message because you are subscribed to the Google Groups "CAS Community" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]<mailto:[email protected]>. To post to this group, send email to [email protected]<mailto:[email protected]>. Visit this group at https://groups.google.com/a/apereo.org/group/cas-user/. To view this discussion on the web visit https://groups.google.com/a/apereo.org/d/msgid/cas-user/BY2PR02MB13327E398AF1E8FAE27F76B1CF6C0%40BY2PR02MB1332.namprd02.prod.outlook.com<https://groups.google.com/a/apereo.org/d/msgid/cas-user/BY2PR02MB13327E398AF1E8FAE27F76B1CF6C0%40BY2PR02MB1332.namprd02.prod.outlook.com?utm_medium=email&utm_source=footer>. For more options, visit https://groups.google.com/a/apereo.org/d/optout. -- You received this message because you are subscribed to the Google Groups "CAS Community" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]<mailto:[email protected]>. To post to this group, send email to [email protected]<mailto:[email protected]>. Visit this group at https://groups.google.com/a/apereo.org/group/cas-user/. To view this discussion on the web visit https://groups.google.com/a/apereo.org/d/msgid/cas-user/C3AE6082-C670-48F0-AE7B-D92C2BFEEC22%40nau.edu<https://groups.google.com/a/apereo.org/d/msgid/cas-user/C3AE6082-C670-48F0-AE7B-D92C2BFEEC22%40nau.edu?utm_medium=email&utm_source=footer>. For more options, visit https://groups.google.com/a/apereo.org/d/optout. -- You received this message because you are subscribed to the Google Groups "CAS Community" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]<mailto:[email protected]>. To post to this group, send email to [email protected]<mailto:[email protected]>. Visit this group at https://groups.google.com/a/apereo.org/group/cas-user/. To view this discussion on the web visit https://groups.google.com/a/apereo.org/d/msgid/cas-user/9d1c5ac8.0000289c.00000002%40MMOAYYED.unicon.net<https://groups.google.com/a/apereo.org/d/msgid/cas-user/9d1c5ac8.0000289c.00000002%40MMOAYYED.unicon.net?utm_medium=email&utm_source=footer>. For more options, visit https://groups.google.com/a/apereo.org/d/optout. -- You received this message because you are subscribed to the Google Groups "CAS Community" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]<mailto:[email protected]>. To post to this group, send email to [email protected]<mailto:[email protected]>. Visit this group at https://groups.google.com/a/apereo.org/group/cas-user/. To view this discussion on the web visit https://groups.google.com/a/apereo.org/d/msgid/cas-user/BY2PR02MB1332B9E835262C255AF5AE99CF6D0%40BY2PR02MB1332.namprd02.prod.outlook.com<https://groups.google.com/a/apereo.org/d/msgid/cas-user/BY2PR02MB1332B9E835262C255AF5AE99CF6D0%40BY2PR02MB1332.namprd02.prod.outlook.com?utm_medium=email&utm_source=footer>. For more options, visit https://groups.google.com/a/apereo.org/d/optout. -- You received this message because you are subscribed to the Google Groups "CAS Community" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/a/apereo.org/group/cas-user/. To view this discussion on the web visit https://groups.google.com/a/apereo.org/d/msgid/cas-user/BY2PR02MB13326BA7F3F833F4CEC190D1CF6D0%40BY2PR02MB1332.namprd02.prod.outlook.com. For more options, visit https://groups.google.com/a/apereo.org/d/optout.
