This kind of problem doesn't look obvious at all... I never thought I would have to digg into mina's internal! ;.(
>From what I understand, ProxyA locks on Context of ProxyB ProxyB locks on Context of ProxyA (The synchronize block) But I don't understand ThreadLocal and how come it blocks on LinkedList<Event> eventQueue = eventQueueThreadLocal.get(); What does ProxyA holds that ProxyB wants? I don't know if I can be of much help, I'am a bit new to this whole concurrency thing and to Mina's StateMachine's internals... But I'll give it a shy try (because this goes into a commercial product! ...) Good luck to us both, Simon -----Original Message----- From: Niklas Therning [mailto:[EMAIL PROTECTED] Sent: May-30-08 2:25 AM To: [email protected] Subject: Re: StateMachine and session.write may block Sorry for not responding any sooner. I had a look at your thread dump and there's no doubt that there's a potential for deadlocks in StateMachine when multiple threads make calls to the same state machine. If we could just find a way to replace that synchronization with something which won't deadlock. I'm not sure how to do that. If threadA is processing events and threadB calls a method on the proxy which generates a new event one solution would be that the new event is added to the event queue and handled by threadA instead of threadB. This means that threadB won't block at all in that case. However, I'm not certain whether that would be the most intuitive thing to do. I think most users would expect threadB to block until the event has been processed. On the other hand, this is already how StateMachine handles recursive events. I.e. when threadA generates a new event while already processing an event. In that case threadA won't block on the method call which generates the new event. If it would block it would of course deadlock itself. Anyways, any suggestions on how to fix the problem described above are very much appreciated. BTW, I thought it should work if you moved the executorFilter before the two state machine filters. I find that quite strange. I thought the executor filter ensured that only one thread is processing MINA events for a particular IoSession at a time. Maybe someone else can shine some light on this issue. Regards, Niklas Simon Trudeau skrev: > Sorry to get back to you on this, but did you find anything interesting? > This issue is a bit of a show stopper for me as I seem to be getting > it quite often. Anything that you could recommend? I tried moving the > executor around but to no avail, I also tried putting a dummy filter > between the two state machines but that didn't solved the issue. I > guess I am only left off with the option of removing the > ExecutorFilter all together. > > Thanks, > > Simon > -----Original Message----- > From: Niklas Therning [mailto:[EMAIL PROTECTED] > Sent: May-27-08 2:58 AM > To: [email protected] > Subject: Re: StateMachine and session.write may block > > Sorry, you're right I already got the attachment. :-[ Will have a look > > at it later today. > > /Niklas > > Simon Trudeau skrev: > >> Didn't you get the thread dump as an attachment to my previous >> > message? > >> A file named deadlock - Copy.txt. If not, let me know how can I send >> you attachments. >> >> I am pretty sure that moving the executorFilter right after My codec >> Filter would solve the issue since both statemachine would execute >> inside the same thread. >> >> >> Simon >> -----Original Message----- >> From: Niklas Therning [mailto:[EMAIL PROTECTED] >> Sent: May-26-08 10:58 AM >> To: [email protected] >> Subject: Re: StateMachine and session.write may block >> >> This looks weird! :) Could you post the complete thread dump? The two >> state machine filters shouldn't lock each other up since they use >> seperate StateContext instances. It's probably got to do with the >> ExecutorFilter. Could you try to move it before the StateMachine >> filters in your chain and see if that helps? >> >> /Niklas >> >> Simon Trudeau skrev: >> >> >>> Found a deadlock! Jstack is a very cool tool! :.) >>> >>> It seams that the problem lies with: >>> >>> ********** >>> StateMachine.java L.130 >>> >>> public void handle(Event event) { >>> StateContext context = event.getContext(); >>> >>> synchronized (context) { >>> LinkedList<Event> eventQueue = >>> eventQueueThreadLocal.get(); ... >>> ********** >>> >>> I am not sure as to why there would be contention between my two >>> statemachine's context (FirmwareUpgradeFilter$FirmwareUpgradeContext >>> and PasswordProtection$PasswordProtectionContext). My only guess is >>> that since both state machines are executed one after the other: >>> >>> 1- My Codec Filter >>> 2- StateMachine Filter 1 (Mina state-machine) >>> 3- StateMachine Filter 2 (Mina state-machine) >>> 4- executorFilter >>> 5- My Processor Filter >>> >>> One filter must be attempting to write a request while the other one >>> must be attempting to write a response and they both lock on the >>> other's resource. >>> >>> "NioProcessor-1": >>> at $Proxy46.messageSent(Unknown Source) >>> >>> "pool-5-thread-1": >>> at $Proxy46.filterWrite(Unknown Source) >>> >>> That's my guess. What do you think? Should this be addressed at the >>> framework level or should I but a dummy filter between my two state >>> machines to make sure that both statemachines don't compete for the >>> same locked resource? Should I change something to my executor >>> > filter? > >>> Thanks, >>> >>> Simon >>> -----Original Message----- >>> From: Niklas Therning [mailto:[EMAIL PROTECTED] >>> Sent: May-26-08 2:14 AM >>> To: [email protected] >>> Subject: Re: StateMachine and session.write may block >>> >>> Hi Simon, >>> >>> I think a thread dump would help debugging this. Please generate one >>> the next time you experience this problem and post the output here. >>> If >>> >>> >> >> >>> you don't know how to generate one please see: >>> http://wiki.netbeans.org/GenerateThreadDump >>> >>> /Niklas >>> >>> Simon Trudeau skrev: >>> >>> >>> >>>> I think I have uncovered a very weird behavior of mina 2.x (latest >>>> trunk). >>>> >>>> I have a filterchain structure as followed >>>> >>>> 1- My Codec Filter >>>> 2- StateMachine Filter 1 (Mina state-machine) >>>> 3- StateMachine Filter 2 (Mina state-machine) >>>> 4- executorFilter >>>> 5- My Processor Filter >>>> >>>> On certain usage scenarios, calls to session.write() block! >>>> >>>> It blocks when DefaultIoFilterChain.callPreviousFilterWrite(){ ... >>>> filterWrite(nf, session, writeRequest); ...} is invoked. >>>> When trying to print the name of the filter that gets invoked >>>> (filter.getClass.getSimpleName()), I get $Proxy46. I would have >>>> wanted >>>> >>>> >>>> >>> >>> >>> >>>> to be more precise and put a break point to know what's going on >>>> but >>>> > > >>>> since I can only reproduce the scenario 1/10 try, it's a bit hard >>>> :.) >>>> >>>> Nevertheless I got the following order of filters called when it >>>> blocked: >>>> org.apache.mina.common.DefaultIoFilterChain$TailFilter >>>> >>>> foo.bar.MyProcessorFilter >>>> >>>> org.apache.mina.filter.executor.ExecutorFilter >>>> >>>> $Proxy46 >>>> >>>> $Proxy46 >>>> >>>> >>>> I cannot explain why it could have blocked since I don't know where >>>> to >>>> >>>> >>>> >>> >>> >>> >>>> start in the Proxy and in Mina StateMachine. Maybe it has to do >>>> with >>>> > > >>>> my ExcutorFilter being a >>>> org.springframework.scheduling.concurrent.ThreadPoolTaskExecutor >>>> instead of an OrderedThreadPoolExecutor >>>> <http://mina.apache.org/report/trunk/apidocs/org/apache/mina/filter >>>> / e x ec utor/OrderedThreadPoolExecutor.html> . >>>> >>>> I would really appreciate any help you can give me on the matter >>>> since >>>> >>>> >>>> >>> >>> >>> >>>> this application I am building will go commercial soon. Thanks. >>>> >>>> >>>> Simon >>>> >>>> >>>> >>>> >>>> >>> >>> >>> >> >> > >
