Hi David! We have CAS 4.0.0, also with Eh-Cache-based ticket registry, on a 4-node cluster. Our configuration of EhCache is almost identical to yours.
Two weeks after our initial deployment, we started getting OOME too, on all nodes. Our system admin measured heap consumption and the resulting graphs show that it is constantly growing until an OOME is thrown out. We gathered a memory snapshot and it showed that majority of the heap was occupied by tickets. I switched on a ticket registry cleaner job in ticketRegistry.xml and scheduled it to run every hour: <bean id="triggerJobDetailTicketRegistryCleaner" class="org.springframework.scheduling.quartz.SimpleTriggerBean" p:jobDetail-ref="jobDetailTicketRegistryCleaner" p:startDelay="20000" p:repeatInterval="3600000" /> The documentation at http://jasig.github.io/cas/4.0.0/installation/Ehcache-Ticket-Registry.html says that the cleaner is not necessary when you use EhCache. Now I'm not sure if I can trust it or not. To be sure, I will keep the cleaner active. Do you have the cleaner enabled or not? We are going to perform a test that should show if tickets are cleaned or not. I have also found that EhCache is able to limit the heap memory consumed by its caches: http://ehcache.org/generated/2.9.0/html/ehc-all/#page/Ehcache_Documentation_Set%2Fco-size_sizing_attributes.html%23 So I tried the following in ehcache-replicated.xml: <ehcache name="ehCacheTicketRegistryCache" updateCheck="false" maxBytesLocalHeap="256M" maxBytesLocalDisk="10G" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; xsi:noNamespaceSchemaLocation="http://ehcache.org/ehcache.xsd";> Unfortunately, it does not work together with Spring's EhCache support used by CAS. EhCacheFactoryBean always provides a limit of the number of elements (even if we do not specify it), which clashes with the heap memory limit and an error is thrown out on startup. In order to use the heap memory limit, we would have to provide a replacement of EhCacheFactoryBean. Best Regards, Jarda -----Original Message----- From: David A. Kovacic [mailto:[email protected]] Sent: 14. November 2014 3:30 odp. To: [email protected] Subject: [cas-user] CAS 4.0.0 Production Issue: Heap Memory Issue All, For the the second time both of our SSO servers running under Tomcat ran out of heap memory last night. They had been up about 7 days straight with no restarts. It looks like they again ran out of memory at about 1GB used (which seems to be the default Java heap size). We have lots of memory available on those servers so the last time this happened, we thought to increase the max heap size to 2GB. Our research had indicated that to increase heap memory for a Java app running under Tomcat you need to add the following line in the Tomcat CATALINA_HOME/bin/setenv.sh file: CATALINA_OPTS=-Xms1000m -Xmx2000m Supposedly according to our research, this increases minimum heap size to 1000MB and max heap size to 2000MB (just under 1GB and 2GB respectively). This is all running under RHEL 6 with Tomcat 7.0.54 and Oracle Java jdk1.8.0_05. Is there something we are missing here? Do we need to do something to tell Tomcat that it needs to allocate more memory than the default to the CAS application itself? The only applications we are running under Tomcat are the CAS webapp and the CAS management webapp which is pretty much idle all the time. We relaod services using the default 2 minute timer in both CAS and CAS-management. This is a fairly major issue for us as we are in the middle of our student registration period and we are seeing huge usage from Blackboard during the late-night hours (which is perversely when these servers tend to run out of heap). People are beginning to take a very jaundiced view of the supposedly "improved" SSO service that our move from RubyCAS was supposed to give them. Dave -- You are currently subscribed to [email protected] as: [email protected] To unsubscribe, change settings or access archives, see http://www.ja-sig.org/wiki/display/JSG/cas-user Join IDC beginning October 29, 2014 through January 29, 2015 for: IDC's 2015 Predictions and IDC FutureScapes Web Conference Series<www.idc.com/predictions2015> Accelerating Innovation on the 3rd Platform Register Now<http://event.on24.com/r.htm?e=861361&s=1&k=223AFC21785863D975C9D80CEE2A97C2> -- You are currently subscribed to [email protected] as: [email protected] To unsubscribe, change settings or access archives, see http://www.ja-sig.org/wiki/display/JSG/cas-user
