Yan, This looks like a race condition between ST validation and memcached replication. We had this issue with 3.5.2.1 and ehcache. The service would try to validate the ticket before ehcache could replicate it. This happened even with a sticky load balancer. Perhaps you could turn on memcached logging in the CAS logs to see when replication and validation are occurring (as well as which server is handling the request). Apache's 1 sec log resolution is not fine enough.
Ray On 2016-05-17 10:44, Yan Zhou wrote: > Hello, > > We are experiencing intermittent ticket error issue with CAS 4.1.7 > overlay setup. The same issue exists in our app based on CAS 3.1.5. I > am not saying that is JASIG CAS issue, most likely it is something in > our configuration. But I cannot find out why. > > We have two servers running CAS on active-active setup with load > balancer setup for session affinity. We have one memcached instance > running on each CAS server, thus two memcached instance running along > with two CAS servers. > > Intermittently we see that /serviceValidate fail when validating > service ticket, because MemcachedTicketRegistry failed to fetch the > ticket. I do not think that is a memcached issue, because it works > fine if I shutdown on CAS server, leaving one single CAS server > running to handle all the traffic. This is why I am not posting this > on memcached mailing list. > > But I do not understand why looking up ticket when we have two CAS > servers running would fail intermittently in MemcachedTicketRegistry. > What I did notice is that, when it fails, it is usually Server01 is > looking up the ticket stored on memcached instance of Server02, or > Server02 is looking up ticket stored on memcached instance on Server 01. > > Any suggestions? > > Thx! > Yan > > > This is my memcached configuration, both servers have identical > setting as follows. > > memcached.servers=server01.dev.medplus.com:11211,server02.dev.medplus.com:11211 > memcached.hashAlgorithm=FNV1_64_HASH > memcached.protocol=BINARY > memcached.locatorType=ARRAY_MOD > memcached.failureMode=Redistribute > > > This is the log I see when it failed. > > On server02 > > > > 172.18.4.136 - - [16/May/2016:20:28:47 +0000] "POST > /cas/login?service=https%3A%2F%2Fintcas.dev.medplus.com%2Fcas-admin%2Fj_spring_cas_security_check > HTTP/1.1" 302 - > > 172.18.4.136 - - [16/May/2016:20:28:49 +0000] "GET > /cas-admin/j_spring_cas_security_check?ticket=ST-1-WLE4H2PcgDuff51TUYnG-dcasde02.dev.medplus.com > HTTP/1.1" 302 – > > > > This is seen on server01 (request now is directed to server01) > > > > 172.18.4.136 - - [16/May/2016:20:28:49 +0000] "GET > /cas/serviceValidate?ticket=ST-1-WLE4H2PcgDuff51TUYnG-dcasde02.dev.medplus.com&service=https%3A%2F%2Fintcas.dev.medplus.com%2Fcas-admin%2Fj_spring_cas_security_check > HTTP/1.1" 200 271 > > > > Cannot find this ticket, therefore, goes back to /login page. If > ticket was found, it should redirect to the App's landing page. But it > does not. > > > > 172.18.4.136 - - [16/May/2016:20:28:49 +0000] "GET > /cas/login?service=https%3A%2F%2Fintcas.dev.medplus.com%2Fcas-admin%2Fj_spring_cas_security_check > HTTP/1.1" 302 - > > 172.18.4.136 - - [16/May/2016:20:28:49 +0000] "GET > /cas/serviceValidate?ticket=ST-2-CHAHXB1PAlYxUZ5Ybcu0-dcasde02.dev.medplus.com&service=https%3A%2F%2Fintcas.dev.medplus.com%2Fcas-admin%2Fj_spring_cas_security_check > HTTP/1.1" 200 213 > > -- > You received this message because you are subscribed to the Google > Groups "CAS Community" group. > To unsubscribe from this group and stop receiving emails from it, send > an email to [email protected] > <mailto:[email protected]>. > To post to this group, send email to [email protected] > <mailto:[email protected]>. > Visit this group at > https://groups.google.com/a/apereo.org/group/cas-user/. > To view this discussion on the web visit > https://groups.google.com/a/apereo.org/d/msgid/cas-user/bffc1502-6907-4381-be4e-a95bf7e52381%40apereo.org > <https://groups.google.com/a/apereo.org/d/msgid/cas-user/bffc1502-6907-4381-be4e-a95bf7e52381%40apereo.org?utm_medium=email&utm_source=footer>. > For more options, visit https://groups.google.com/a/apereo.org/d/optout. -- You received this message because you are subscribed to the Google Groups "CAS Community" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/a/apereo.org/group/cas-user/. To view this discussion on the web visit https://groups.google.com/a/apereo.org/d/msgid/cas-user/18192a83-c675-8e35-642b-637cb0545644%40uvic.ca. For more options, visit https://groups.google.com/a/apereo.org/d/optout.
