Hi Scott, Sorry for the slow response; I was trying to get more info before I responded.
The difficult part about debugging this issue is that there aren't any error or warning messages in cas.log. A symptom of this issue is actually a decrease in logging, as the decrement counter for throttling, reload of services from the DB, and ticket cleaner all stop functioning (and thus stop logging). I'm not sure what part of our config is of interest. We're running CAS 3.4.6 with the clearPass extension, and use Inspektr for audit trail logging and login throttling. We use an Oracle database for storing the Inspektr audit trail, JPA ticket registry, and service registry. We run 2 CAS servers for each environment hitting one backend Oracle database. For example Prod CAS server 1 and Prod CAS server 2 are using our prod Oracle DB, and Dev CAS server 1 and Dev CAS server 2 are using our dev Oracle DB. In each environment we have them in a primary/secondary configuration, whereby the primary server services all requests and the other is a hot stand-by (both cleans the ticket registry). When one of our CAS server stops functioning properly, the other one in the environment continues to work properly. This leads me to believe it's not a DB issue. If I then cycle tomcat on the misbehaving server, it functions properly again. I just got the JMX connector tuned up on the CAS servers, so hopefully next time this happens I can debug more thoroughly. Thanks for any help. Matt -- Matthew Selwood [email protected] 250-472-5565 On 03/15/2012 10:36 AM, Scott Battaglia wrote: > I've never seen that before. Can you post any relevant parts of your > config/logs? > > Thanks > Scott > > > On Thu, Mar 15, 2012 at 12:51 PM, Matthew Selwood <[email protected] > <mailto:[email protected]>> wrote: > > Howdy, > > Has anybody had issues with the schedule_Worker-# threads > deadlocking? I suspect this might be my issue because when the > quartz triggered decrementCounts stops (part of login throttling), > the ticket cleaning and reloading from the service registry also > stop functioning. All three tasks are performed by > schedule_Worker-# threads. > > Our tomcat uses the following defaults: > maxThreads default 200 > minSpareThreads default 25 > maxIdleTime default 1 min > > I notice the thread count for our CAS instance is quite high. I > just cycled the instance and it has 125 threads in the process > already. It eventually approaches the limit of 200. > > I'm getting the JMX connector installed so I can use jconsole to > debug further and confirm a deadlock is occurring. > > Any further debugging tips or info on related bugs is appreciated. > > Thanks, > Matt > > > On 03/07/2012 08:20 AM, Matthew Selwood wrote: > > Any thoughts from anybody? All suggestions are appreciated. > > Thanks, > Matt > ________________________________________ > From: Matthew Selwood [[email protected] <mailto:[email protected]>] > Sent: March 5, 2012 12:08 PM > To:[email protected] <mailto:to%[email protected]> > Subject: [cas-user] throttling login attempts /w Inspektr issue > > Howdy, > > I'm currently running CAS 3.4.6 with login throttling using > Inspektr. > > I'm having a problem where people are getting permanently > throttled. It > looks like the decrementCounts() function isn't being called > as I don't > see the log entry (should be something like this: > > "support.InMemoryThrottledSubmissionByIpAddressAndUsernameHandlerInterceptorAdapter.[] > - Decrementing counts for throttler. Starting key count: 0"). > There > are no errors thrown in the cas.log, so I'm a bit lost at how > to debug > this issue. If I restart the CAS server it starts logged the > decrement > counts in cas.log and working properly. > > Has anybody seen this problem before or have any thoughts? > > Thanks, > Matt > > -- > You are currently subscribed [email protected] > <mailto:[email protected]> as:[email protected] > <mailto:as%[email protected]> > To unsubscribe, change settings or access archives, > seehttp://www.ja-sig.org/wiki/display/JSG/cas-user > <http://www.ja-sig.org/wiki/display/JSG/cas-user> > > > -- > You are currently subscribed to [email protected] > <mailto:[email protected]> as: [email protected] > <mailto:[email protected]> > To unsubscribe, change settings or access archives, see > http://www.ja-sig.org/wiki/display/JSG/cas-user > > > -- > You are currently subscribed to [email protected] as: [email protected] > To unsubscribe, change settings or access archives, see > http://www.ja-sig.org/wiki/display/JSG/cas-user -- You are currently subscribed to [email protected] as: [email protected] To unsubscribe, change settings or access archives, see http://www.ja-sig.org/wiki/display/JSG/cas-user
