Sorry for not responding any sooner. I had a look at your thread dump
and there's no doubt that there's a potential for deadlocks in
StateMachine when multiple threads make calls to the same state machine.
If we could just find a way to replace that synchronization with
something which won't deadlock. I'm not sure how to do that.
If threadA is processing events and threadB calls a method on the proxy
which generates a new event one solution would be that the new event is
added to the event queue and handled by threadA instead of threadB. This
means that threadB won't block at all in that case. However, I'm not
certain whether that would be the most intuitive thing to do. I think
most users would expect threadB to block until the event has been
processed. On the other hand, this is already how StateMachine handles
recursive events. I.e. when threadA generates a new event while already
processing an event. In that case threadA won't block on the method call
which generates the new event. If it would block it would of course
deadlock itself.
Anyways, any suggestions on how to fix the problem described above are
very much appreciated.
BTW, I thought it should work if you moved the executorFilter before the
two state machine filters. I find that quite strange. I thought the
executor filter ensured that only one thread is processing MINA events
for a particular IoSession at a time. Maybe someone else can shine some
light on this issue.
Regards,
Niklas
Simon Trudeau skrev:
Sorry to get back to you on this, but did you find anything interesting?
This issue is a bit of a show stopper for me as I seem to be getting it
quite often. Anything that you could recommend? I tried moving the
executor around but to no avail, I also tried putting a dummy filter
between the two state machines but that didn't solved the issue. I guess
I am only left off with the option of removing the ExecutorFilter all
together.
Thanks,
Simon
-----Original Message-----
From: Niklas Therning [mailto:[EMAIL PROTECTED]
Sent: May-27-08 2:58 AM
To: [email protected]
Subject: Re: StateMachine and session.write may block
Sorry, you're right I already got the attachment. :-[ Will have a look
at it later today.
/Niklas
Simon Trudeau skrev:
Didn't you get the thread dump as an attachment to my previous
message?
A file named deadlock - Copy.txt. If not, let me know how can I send
you attachments.
I am pretty sure that moving the executorFilter right after My codec
Filter would solve the issue since both statemachine would execute
inside the same thread.
Simon
-----Original Message-----
From: Niklas Therning [mailto:[EMAIL PROTECTED]
Sent: May-26-08 10:58 AM
To: [email protected]
Subject: Re: StateMachine and session.write may block
This looks weird! :) Could you post the complete thread dump? The two
state machine filters shouldn't lock each other up since they use
seperate StateContext instances. It's probably got to do with the
ExecutorFilter. Could you try to move it before the StateMachine
filters in your chain and see if that helps?
/Niklas
Simon Trudeau skrev:
Found a deadlock! Jstack is a very cool tool! :.)
It seams that the problem lies with:
**********
StateMachine.java L.130
public void handle(Event event) {
StateContext context = event.getContext();
synchronized (context) {
LinkedList<Event> eventQueue =
eventQueueThreadLocal.get(); ...
**********
I am not sure as to why there would be contention between my two
statemachine's context (FirmwareUpgradeFilter$FirmwareUpgradeContext
and PasswordProtection$PasswordProtectionContext). My only guess is
that since both state machines are executed one after the other:
1- My Codec Filter
2- StateMachine Filter 1 (Mina state-machine)
3- StateMachine Filter 2 (Mina state-machine)
4- executorFilter
5- My Processor Filter
One filter must be attempting to write a request while the other one
must be attempting to write a response and they both lock on the
other's resource.
"NioProcessor-1":
at $Proxy46.messageSent(Unknown Source)
"pool-5-thread-1":
at $Proxy46.filterWrite(Unknown Source)
That's my guess. What do you think? Should this be addressed at the
framework level or should I but a dummy filter between my two state
machines to make sure that both statemachines don't compete for the
same locked resource? Should I change something to my executor
filter?
Thanks,
Simon
-----Original Message-----
From: Niklas Therning [mailto:[EMAIL PROTECTED]
Sent: May-26-08 2:14 AM
To: [email protected]
Subject: Re: StateMachine and session.write may block
Hi Simon,
I think a thread dump would help debugging this. Please generate one
the next time you experience this problem and post the output here.
If
you don't know how to generate one please see:
http://wiki.netbeans.org/GenerateThreadDump
/Niklas
Simon Trudeau skrev:
I think I have uncovered a very weird behavior of mina 2.x (latest
trunk).
I have a filterchain structure as followed
1- My Codec Filter
2- StateMachine Filter 1 (Mina state-machine)
3- StateMachine Filter 2 (Mina state-machine)
4- executorFilter
5- My Processor Filter
On certain usage scenarios, calls to session.write() block!
It blocks when DefaultIoFilterChain.callPreviousFilterWrite(){ ...
filterWrite(nf, session, writeRequest); ...} is invoked.
When trying to print the name of the filter that gets invoked
(filter.getClass.getSimpleName()), I get $Proxy46. I would have
wanted
to be more precise and put a break point to know what's going on but
since I can only reproduce the scenario 1/10 try, it's a bit hard
:.)
Nevertheless I got the following order of filters called when it
blocked:
org.apache.mina.common.DefaultIoFilterChain$TailFilter
foo.bar.MyProcessorFilter
org.apache.mina.filter.executor.ExecutorFilter
$Proxy46
$Proxy46
I cannot explain why it could have blocked since I don't know where
to
start in the Proxy and in Mina StateMachine. Maybe it has to do with
my ExcutorFilter being a
org.springframework.scheduling.concurrent.ThreadPoolTaskExecutor
instead of an OrderedThreadPoolExecutor
<http://mina.apache.org/report/trunk/apidocs/org/apache/mina/filter/
e x ec utor/OrderedThreadPoolExecutor.html> .
I would really appreciate any help you can give me on the matter
since
this application I am building will go commercial soon. Thanks.
Simon
