I remembered a similar issue that was filed back a while ago.
See if the discussion here helps: https://github.com/Jasig/cas/pull/297 From: David A. Kovacic [mailto:[email protected]] Sent: Friday, November 14, 2014 1:56 PM To: [email protected] Subject: Re: [cas-user] CAS 4.0.0 Production Issue: Heap Memory Issue Here's my EHCACHE configuration (ticketRegistry.xml): <?xml version="1.0" encoding="UTF-8"?> <beans xmlns= <http://www.springframework.org/schema/beans> "http://www.springframework.org/schema/beans"; xmlns:xsi= <http://www.w3.org/2001/XMLSchema-instance> "http://www.w3.org/2001/XMLSchema-instance"; xmlns:util= <http://www.springframework.org/schema/util> "http://www.springframework.org/schema/util"; xmlns:p= <http://www.springframework.org/schema/p> "http://www.springframework.org/schema/p"; xmlns:c= <http://www.springframework.org/schema/c> "http://www.springframework.org/schema/c"; xsi:schemaLocation=" http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-3.0.xsd http://www.springframework.org/schema/util http://www.springframework.org/schema/util/spring-util-3.0.xsd";> <bean id="ticketRegistry" class="org.jasig.cas.ticket.registry.EhCacheTicketRegistry" p:serviceTicketsCache-ref="serviceTicketsCache" p:ticketGrantingTicketsCache-ref="ticketGrantingTicketsCache" /> <bean id="abstractTicketCache" abstract="true" class="org.springframework.cache.ehcache.EhCacheFactoryBean" p:cacheManager-ref="cacheManager" p:diskExpiryThreadIntervalSeconds="0" p:diskPersistent="false" p:eternal="false" p:maxElementsInMemory="10000" p:maxElementsOnDisk="20000" p:memoryStoreEvictionPolicy="LRU" p:overflowToDisk="true" p:bootstrapCacheLoader-ref="ticketCacheBootstrapCacheLoader" /> <!-- MUST use synchronous repl for service tickets for correct behavior. --> <bean id="serviceTicketsCache" class="org.springframework.cache.ehcache.EhCacheFactoryBean" parent="abstractTicketCache" p:cacheName="cas_st" p:timeToIdle="0" p:timeToLive="300" p:cacheEventListeners-ref="ticketRMISynchronousCacheReplicator" /> <bean id="ticketGrantingTicketsCache" class="org.springframework.cache.ehcache.EhCacheFactoryBean" parent="abstractTicketCache" p:cacheName="cas_tgt" p:timeToIdle="0" p:timeToLive="48801" p:cacheEventListeners-ref="ticketRMIAsynchronousCacheReplicator" /> <bean id="cacheManager" class="org.springframework.cache.ehcache.EhCacheManagerFactoryBean" p:configLocation="classpath:ehcache-replicated.xml" p:shared="false" p:cacheManagerName="ticketRegistryCacheManager" /> <bean id="ticketRMISynchronousCacheReplicator" class="net.sf.ehcache.distribution.RMISynchronousCacheReplicator" c:replicatePuts="true" c:replicatePutsViaCopy="true" c:replicateUpdates="true" c:replicateUpdatesViaCopy="true" c:replicateRemovals="true" /> <bean id="ticketRMIAsynchronousCacheReplicator" class="net.sf.ehcache.distribution.RMIAsynchronousCacheReplicator" parent="ticketRMISynchronousCacheReplicator" c:replicationInterval="10000" c:maximumBatchSize="100" /> <bean id="ticketCacheBootstrapCacheLoader" class="net.sf.ehcache.distribution.RMIBootstrapCacheLoader" c:asynchronous="true" c:maximumChunkSize="5000000" /> </beans> The ticket expiration policy is just what came with CAS and the CAS properties files are using # Defaults sourced from WEB-INF/spring-configuration/ticketExpirationPolices.xml # # Maximum session timeout - TGT will expire in maxTimeToLiveInSeconds regardless of usage tgt.maxTimeToLiveInSeconds=86400 # # Idle session timeout - TGT will expire sooner than maxTimeToLiveInSeconds if no further requests # for STs occur within timeToKillInSeconds tgt.timeToKillInSeconds=21600 ## # Service Ticket Timeout # Default sourced from WEB-INF/spring-configuration/ticketExpirationPolices.xml # # Service Ticket timeout - typically kept short as a control against replay attacks, default is 10s. You'll want to # increase this timeout if you are manually testing service ticket creation/validation via tamperdata or similar tools st.timeToKillInSeconds=30 On 11/14/14 3:08 PM, Scott Battaglia wrote: What is your eviction policy for things in the cache? On Fri Nov 14 2014 at 3:06:16 PM David A. Kovacic <[email protected]> wrote: Do those setting go in CATALINA_OPTS or JAVA_OPTS in the setenv.sh file? On 11/14/14 9:39 AM, Zac Harvey wrote: > We've been using 2 load balanced 4.0.0 nodes (LDAP auth handler) for > almost 3 months now and never needed a restart. That tells me this is > either specific to your configuration or custom code (3rd party of > inhouse) you've added on top of what CAS provides. Either way it doesn't > seem to be a problem with CAS core. Can you give us more details about > your setup? > > Most importantly, you will want to profile your server with something like > jVisualVM (comes with the JDK). This will help you see what is actually > going on with your memory. > > http://visualvm.java.net/gettingstarted.html > > Even more most importantly, you can configure your Tomcat nodes to dump > the memory heap to a file when your experience on OutOfMemoryException: > > -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=<path to dump file> > > If this happens again in PROD, or if you can reproduce it in non-prod, > load one of these files in jVisualVM and explore its contents. Without > too much poking around it should quickly tell you where most of your > memory was getting chewed up. If you report back with your findings from > this heap dump analysis we can better pinpoint where your memory leak is > coming from. > > HTH > ________________________________________ > From: David A. Kovacic <[email protected]> > Sent: Friday, November 14, 2014 9:30 AM > To: [email protected] > Subject: [cas-user] CAS 4.0.0 Production Issue: Heap Memory Issue > > All, > > For the the second time both of our SSO servers running under Tomcat ran > out of heap memory last night. They had been up about 7 days straight > with no restarts. It looks like they again ran out of memory at about > 1GB used (which seems to be the default Java heap size). We have lots > of memory available on those servers so the last time this happened, we > thought to increase the max heap size to 2GB. Our research had > indicated that to increase heap memory for a Java app running under > Tomcat you need to add the following line in the Tomcat > CATALINA_HOME/bin/setenv.sh file: > > CATALINA_OPTS=-Xms1000m -Xmx2000m > > Supposedly according to our research, this increases minimum heap size > to 1000MB and max heap size to 2000MB (just under 1GB and 2GB > respectively). This is all running under RHEL 6 with Tomcat 7.0.54 and > Oracle Java jdk1.8.0_05. Is there something we are missing here? Do we > need to do something to tell Tomcat that it needs to allocate more > memory than the default to the CAS application itself? The only > applications we are running under Tomcat are the CAS webapp and the CAS > management webapp which is pretty much idle all the time. We relaod > services using the default 2 minute timer in both CAS and CAS-management. > > This is a fairly major issue for us as we are in the middle of our > student registration period and we are seeing huge usage from Blackboard > during the late-night hours (which is perversely when these servers tend > to run out of heap). People are beginning to take a very jaundiced view > of the supposedly "improved" SSO service that our move from RubyCAS was > supposed to give them. > > Dave > > > -- > You are currently subscribed to [email protected] as: > [email protected] > To unsubscribe, change settings or access archives, see > http://www.ja-sig.org/wiki/display/JSG/cas-user -- You are currently subscribed to [email protected] as: [email protected] To unsubscribe, change settings or access archives, see http://www.ja-sig.org/wiki/display/JSG/cas-user -- You are currently subscribed to [email protected] as: [email protected] To unsubscribe, change settings or access archives, see http://www.ja-sig.org/wiki/display/JSG/cas-user -- You are currently subscribed to [email protected] as: [email protected] To unsubscribe, change settings or access archives, see http://www.ja-sig.org/wiki/display/JSG/cas-user -- You are currently subscribed to [email protected] as: [email protected] To unsubscribe, change settings or access archives, see http://www.ja-sig.org/wiki/display/JSG/cas-user
