This email I sent looks like it got stuck in Google yesterday for nearly 
2-1/2 hours before delivery (cf. Received lines in mail header). List 
maintainers: Two followup emails I sent yesterday mid-day on this topic 
still have not been delivered.

In summary:

The RegistryCleaner was moved/refactored in 4.2.5 out of the default 
(in-memory) ticket registry to a TicketRegistryCleaner and set to 
auto-start by default on all registry types (including Hazelcast, Ehcache, 
etc).

On deploying 4.2.6, cf. the most recent CAS security announcement, all 
servers in our pool started competing for access to the ticket cache (for 
us, 50k-60k entries), walking the entire cache every two minutes at chunks 
of 500 tickets at a time and using what looks like an exclusive lock.

Disabling the registry cleaner brought load average on our (4) servers back 
down to normal levels (0.01-0.20), a twenty-fold decrease in load average. 
We're currently using Hazelcast, which has its own TTL eviction mechanism, 
so this cleaner is not necessary. The same holds for Ehcache (which we used 
previously).


cas.properties:


ticket.registry.cleaner.startdelay=-1


(value could have been zero, but -1 seemed more mnemonic of the intent)


If someone does need to use this ticket cleaner, it seems to make sense to 
run on only a single node, assuming global cache semantics to the 
get-all-entries method (don't recall the exact name at the moment).


Tom.

On Friday, October 14, 2016 at 10:14:42 AM UTC-7, tfpoage wrote:
>
> Disabling the fourth node doesn't change anything. 
>
> Profiling shows the highest CPU/time is spent in Hazelcast. Whether this 
> is a result of the updated Hazelcast version or the new synchronous CAS 
> code remains to be seen. 
>
> Is it oossible to downgrade Hazelcast version (say, to 3.6) on CAS 4.2.6, 
> i.e. were any new Hazelcast version-specific changes made between roughly 
> 4.2.[12] and 4.2.6? 
>
> Thanks. 
> Tom. 
>
> > On Oct 13, 2016, at 2:18 PM, Tom Poage <tfp...@ucdavis.edu <javascript:>> 
> wrote: 
> > 
> > Afternoon, 
> > 
> > On moving from 4.2.1 to 4.2.6, our apparent system load increased 
> dramatically. 
> > 
> > Run queue went from as high as 4 to nearly 30, with (Linux) load average 
> jumping from a max of 0.2 to about 15 for a user base (TGT count) of 46k. 
> > 
> > A code diff doesn’t seem to show much, except perhaps for the addition 
> of a synchronous ticketTransactionManager. The only other likely candidate 
> is either the bump in Hazelcast version, or that we went from 3 to 4 
> (single CPU) VMs in the cluster (point-to-point instead of multicast). CPU 
> increased from a high of about 20% (usually 5-8%) to the 50% range. This is 
> on all nodes. Ironically, response time doesn’t seem all that bad, though 
> is a bit sluggish. 
> > 
> > Anyone else experience something similar? 
> > 
> > Thanks! 
> > Tom. 
> > 
> > -- 
> > CAS gitter chatroom: https://gitter.im/apereo/cas 
> > CAS mailing list guidelines: 
> https://apereo.github.io/cas/Mailing-Lists.html 
> > CAS documentation website: https://apereo.github.io/cas 
> > CAS project website: https://github.com/apereo/cas 
> > --- 
> > You received this message because you are subscribed to the Google 
> Groups "CAS Community" group. 
> > To unsubscribe from this group and stop receiving emails from it, send 
> an email to cas-user+u...@apereo.org <javascript:>. 
> > To post to this group, send email to cas-...@apereo.org <javascript:>. 
> > Visit this group at 
> https://groups.google.com/a/apereo.org/group/cas-user/. 
> > To view this discussion on the web visit 
> https://groups.google.com/a/apereo.org/d/msgid/cas-user/F67D31AA-2CFC-4DDA-8C5D-922E0B87798F%40ucdavis.edu.
>  
>
> > For more options, visit https://groups.google.com/a/apereo.org/d/optout. 
>
>
>

-- 
CAS gitter chatroom: https://gitter.im/apereo/cas
CAS mailing list guidelines: https://apereo.github.io/cas/Mailing-Lists.html
CAS documentation website: https://apereo.github.io/cas
CAS project website: https://github.com/apereo/cas
--- 
You received this message because you are subscribed to the Google Groups "CAS 
Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to cas-user+unsubscr...@apereo.org.
To post to this group, send email to cas-user@apereo.org.
Visit this group at https://groups.google.com/a/apereo.org/group/cas-user/.
To view this discussion on the web visit 
https://groups.google.com/a/apereo.org/d/msgid/cas-user/fbd62617-ff94-4150-bab7-005405b427b1%40apereo.org.
For more options, visit https://groups.google.com/a/apereo.org/d/optout.

Reply via email to