Status: Accepted
Owner: vladimir.ralev
Labels: Type-Defect Priority-Medium
New issue 1815 by vladimir.ralev: MSS app session timer service is
vulnerable to exceptions freezing the whole server when new app session is
constructed
http://code.google.com/p/mobicents/issues/detail?id=1815
The scheduling of a timer using the standard timer service produces the
folowing excpetion under load
java.lang.IllegalStateException: Timer already cancelled.
at java.util.Timer.sched(Timer.java:354)
at java.util.Timer.schedule(Timer.java:170)
at
org.mobicents.servlet.sip.core.timers.StandardSipApplicationSessionTimerService.schedule(StandardSipApplicationSessionTimerService.java:73)
at
org.mobicents.servlet.sip.core.session.SipApplicationSessionImpl.<init>(SipApplicationSessionImpl.java:145)
at
org.jboss.web.tomcat.service.session.ClusteredSipApplicationSession.<init>(ClusteredSipApplicationSession.java:203)
at
org.jboss.web.tomcat.service.session.JBossCacheClusteredSipApplicationSession.<init>(JBossCacheClusteredSipApplicationSession.java:45)
at
org.jboss.web.tomcat.service.session.SessionBasedClusteredSipApplicationSession.<init>(SessionBasedClusteredSipApplicationSession.java:71)
at
org.jboss.web.tomcat.service.session.ClusteredSipManagerDelegate.getNewMobicentsSipApplicationSession(ClusteredSipManagerDelegate.java:67)
at
org.mobicents.servlet.sip.core.session.SipManagerDelegate.createSipApplicationSession(SipManagerDelegate.java:212)
at
org.mobicents.servlet.sip.core.session.SipManagerDelegate.getSipApplicationSession(SipManagerDelegate.java:204)
at
org.jboss.web.tomcat.service.session.JBossCacheSipManager.loadSipApplicationSession(JBossCacheSipManager.java:1491)
at
org.jboss.web.tomcat.service.session.JBossCacheSipManager.getSipApplicationSession(JBossCacheSipManager.java:2847)
at
org.mobicents.servlet.sip.core.dispatchers.InitialRequestDispatcher.dispatchInsideContainer(InitialRequestDispatcher.java:368)
at
org.mobicents.servlet.sip.core.dispatchers.InitialRequestDispatcher.dispatchMessage(InitialRequestDispatcher.java:288)
at
org.mobicents.servlet.sip.core.SipApplicationDispatcherImpl.processRequest(SipApplicationDispatcherImpl.java:661)
at gov.nist.javax.sip.EventScanner.deliverEvent(EventScanner.java:224)
at
gov.nist.javax.sip.SipProviderImpl.handleEvent(SipProviderImpl.java:192)
at
gov.nist.javax.sip.DialogFilter.processRequest(DialogFilter.java:1151)
at
gov.nist.javax.sip.stack.SIPServerTransaction.processRequest(SIPServerTransaction.java:823)
at
gov.nist.javax.sip.stack.TCPMessageChannel.processMessage(TCPMessageChannel.java:552)
at
gov.nist.javax.sip.parser.PipelinedMsgParser$1.run(PipelinedMsgParser.java:426)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
It seems that the problem is rooted in the way the java timer starts
throwing IllegalStateExceptions when it's thread is compromised for some
reason. This simple app reproduces the condition by killing the thread with
uncought exception (this is likely what happens in MSS too.
public static void main(String argv[]) throws Exception
{
Timer timer = new Timer();
for(int q=0;q<10000;q++) {
TimerTask t =new TimerTask() {
@Override
public void run() {
System.out.print("|");
throw new RuntimeException();
}
};
timer.schedule(t, 0, 1000);
}
}
To solve the problem we should use a ThreadpooledTimer that recovers killed
threads + we should catch the exceptions.
