Hi Emmanuel, I'm not sure if I understand what you're saying or asking. My email client doesn't do a great job of formatting all that code in the email either, which might add to the confusion.
The code that you seem to refer to is here: https://github.com/igniterealtime/Openfire/blob/cd28f39c411a0faede6d04454caf865d1695928f/xmppserver/src/main/java/org/jivesoftware/openfire/session/LocalClientSession.java#L923-L932 There is a lock that intends to ensure that the 'streammanager' (which keeps statistics on data sent, received and acknowledged) is not used while data is being sent over the outbound connection. Does that make sense and answer your question? Kind regards, Guus On Mon, Dec 19, 2022 at 4:00 AM Emmanuel Lécharny <elecha...@gmail.com> wrote: > Hi Guus, > > thanks for teh full stack trace. > > I've some concern about this blocked thread: > "TaskEngine-pool-3" #40 daemon prio=5 waiting on lock > java.lang.Thread.State: BLOCKED > - blocked on org.apache.mina.filter.ssl.SSLHandlerG0@99f49e2 > (owned by > socket_c2s_ssl-thread-2 id=78) > at > org.apache.mina.filter.ssl.SSLHandlerG0.write(SSLHandlerG0.java:312) > at > org.apache.mina.filter.ssl.SslFilter.filterWrite(SslFilter.java:380) > at > > org.apache.mina.core.filterchain.DefaultIoFilterChain.callPreviousFilterWrite(DefaultIoFilterChain.java:753) > at > > org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1500(DefaultIoFilterChain.java:49) > at > > org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.filterWrite(DefaultIoFilterChain.java:1146) > at > > org.apache.mina.filter.codec.ProtocolCodecFilter.filterWrite(ProtocolCodecFilter.java:301) > at > > org.apache.mina.core.filterchain.DefaultIoFilterChain.callPreviousFilterWrite(DefaultIoFilterChain.java:753) > at > > org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1500(DefaultIoFilterChain.java:49) > at > > org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.filterWrite(DefaultIoFilterChain.java:1146) > at > > org.apache.mina.core.filterchain.IoFilterAdapter.filterWrite(IoFilterAdapter.java:138) > at > org.jivesoftware.openfire.net > .StalledSessionsFilter.filterWrite(StalledSessionsFilter.java:61) > at > > org.apache.mina.core.filterchain.DefaultIoFilterChain.callPreviousFilterWrite(DefaultIoFilterChain.java:753) > at > > org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1500(DefaultIoFilterChain.java:49) > at > > org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.filterWrite(DefaultIoFilterChain.java:1146) > at > > org.apache.mina.core.filterchain.IoFilterAdapter.filterWrite(IoFilterAdapter.java:138) > at > > org.apache.mina.core.filterchain.DefaultIoFilterChain.callPreviousFilterWrite(DefaultIoFilterChain.java:753) > at > > org.apache.mina.core.filterchain.DefaultIoFilterChain.fireFilterWrite(DefaultIoFilterChain.java:746) > at > > org.apache.mina.core.session.AbstractIoSession.write(AbstractIoSession.java:575) > at > > org.apache.mina.core.session.AbstractIoSession.write(AbstractIoSession.java:520) > at > org.jivesoftware.openfire.nio.NIOConnection.deliver(NIOConnection.java:349) > at > > org.jivesoftware.openfire.session.LocalClientSession.deliver(LocalClientSession.java:928) > - locked > org.jivesoftware.openfire.streammanagement.StreamManager@7e024771 > at > > org.jivesoftware.openfire.session.LocalSession.process(LocalSession.java:407) > at > > org.jivesoftware.openfire.spi.RoutingTableImpl.routeToLocalDomain(RoutingTableImpl.java:439) > at > > org.jivesoftware.openfire.spi.RoutingTableImpl.routePacket(RoutingTableImpl.java:350) > ... > > > As you can see, there is a lock on a StreamManager instance following a > call to org.jivesoftware.openfire.session.La ocalSession.process: > > public void process(Packet packet) { > // Check that the requested packet can be processed > if (canProcess(packet)) { > // Perform the actual processing of the packet. This > usually implies sending > // the packet to the entity > try { > // Invoke the interceptors before we send the packet > > InterceptorManager.getInstance().invokeInterceptors(packet, this, false, > false); > deliver(packet); > <<<-------------------------------------Here > > but I don't see why a lock is taken, unless I don't have the proper > source code. > > Which version are you using when having this deadlock? > > > On 14/12/2022 09:37, Guus der Kinderen wrote: > > Hi Emmanuel, > > > > I was trying to be helpful by leaving out unimportant details, but I > > seem to have messed up. A third thread is involved. I've now attached > > the full thread dump to this email. > > > > Kind regards, > > > > Guus > > > > > > On Mon, Dec 12, 2022 at 9:53 AM Emmanuel Lécharny <elecha...@gmail.com > > <mailto:elecha...@gmail.com>> wrote: > > > > Hi Guus, > > > > there is something missing in the stacck trace: the two threads are > > blocked on different monitors: > > * SSLHandlerG0@99f49e2 owned by socket_c2s_ssl-thread-2 > > * StreamManager@7e024771 owned by TaskEngine-pool-3 > > > > they are not related, they are not blocking each one other. > > > > Can you provide the full stack trace ? > > > > Thanks ! > > > > > > On 09/12/2022 10:02, Guus der Kinderen wrote: > > > Hello, > > > > > > I wonder if MINA 2.2.1 introduces a new potential for deadlock in > its > > > SSLHandler implementation. Did something change there, as > > compared to MINA > > > 2.1.3? > > > > > > Since this upgrade, we consistently run into a thread deadlock. > > One of the > > > threads that is deadlocking is sending data to the peer, while > > the other is > > > processing data from the peer. The deadlock involves a MINA-based > > lock (in > > > SSLHandler), and a lock in our code (StreamManager), that is > > responsible > > > for recording what packets that are exchanged have been > > acknowledged by the > > > peer. The latter did not change, so I'm suspecting a change in > > SSLHandler. > > > > > > The stack traces of two deadlocked threads are added to this > email. > > > > > > Kind regards, > > > > > > Guus > > > > > > > > > "TaskEngine-pool-3" #40 daemon prio=5 waiting on lock > > > java.lang.Thread.State: BLOCKED > > > - blocked on org.apache.mina.filter.ssl.SSLHandlerG0@99f49e2 > > (owned by > > > socket_c2s_ssl-thread-2 id=78) > > > at > > org.apache.mina.filter.ssl.SSLHandlerG0.write(SSLHandlerG0.java:312) > > > at > > org.apache.mina.filter.ssl.SslFilter.filterWrite(SslFilter.java:380) > > > at > > > > > > > org.apache.mina.core.filterchain.DefaultIoFilterChain.callPreviousFilterWrite(DefaultIoFilterChain.java:753) > > > at > > > > > > > org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1500(DefaultIoFilterChain.java:49) > > > at > > > > > > > org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.filterWrite(DefaultIoFilterChain.java:1146) > > > at > > > > > > > org.apache.mina.filter.codec.ProtocolCodecFilter.filterWrite(ProtocolCodecFilter.java:301) > > > at > > > > > > > org.apache.mina.core.filterchain.DefaultIoFilterChain.callPreviousFilterWrite(DefaultIoFilterChain.java:753) > > > at > > > > > > > org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1500(DefaultIoFilterChain.java:49) > > > at > > > > > > > org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.filterWrite(DefaultIoFilterChain.java:1146) > > > at > > > > > > > org.apache.mina.core.filterchain.IoFilterAdapter.filterWrite(IoFilterAdapter.java:138) > > > at > > > org.jivesoftware.openfire.net > > <http://org.jivesoftware.openfire.net > >.StalledSessionsFilter.filterWrite(StalledSessionsFilter.java:61) > > > at > > > > > > > org.apache.mina.core.filterchain.DefaultIoFilterChain.callPreviousFilterWrite(DefaultIoFilterChain.java:753) > > > at > > > > > > > org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1500(DefaultIoFilterChain.java:49) > > > at > > > > > > > org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.filterWrite(DefaultIoFilterChain.java:1146) > > > at > > > > > > > org.apache.mina.core.filterchain.IoFilterAdapter.filterWrite(IoFilterAdapter.java:138) > > > at > > > > > > > org.apache.mina.core.filterchain.DefaultIoFilterChain.callPreviousFilterWrite(DefaultIoFilterChain.java:753) > > > at > > > > > > > org.apache.mina.core.filterchain.DefaultIoFilterChain.fireFilterWrite(DefaultIoFilterChain.java:746) > > > at > > > > > > > org.apache.mina.core.session.AbstractIoSession.write(AbstractIoSession.java:575) > > > at > > > > > > > org.apache.mina.core.session.AbstractIoSession.write(AbstractIoSession.java:520) > > > at > > > > > > org.jivesoftware.openfire.nio.NIOConnection.deliver(NIOConnection.java:349) > > > at > > > > > > > org.jivesoftware.openfire.session.LocalClientSession.deliver(LocalClientSession.java:928) > > > - locked > > org.jivesoftware.openfire.streammanagement.StreamManager@7e024771 > > > at > > > > > > org.jivesoftware.openfire.session.LocalSession.process(LocalSession.java:407) > > > at > > > > > > > org.jivesoftware.openfire.spi.RoutingTableImpl.routeToLocalDomain(RoutingTableImpl.java:439) > > > at > > > > > > > org.jivesoftware.openfire.spi.RoutingTableImpl.routePacket(RoutingTableImpl.java:350) > > > at org.jivesoftware.openfire.IQRouter.handle(IQRouter.java:426) > > > at org.jivesoftware.openfire.IQRouter.route(IQRouter.java:106) > > > at > > > > > > > org.jivesoftware.openfire.spi.PacketRouterImpl.route(PacketRouterImpl.java:74) > > > at > > > > > > > org.jivesoftware.openfire.pubsub.PubSubEngine.publishItemsToNode(PubSubEngine.java:428) > > > at > > > > > > org.jivesoftware.openfire.pubsub.PubSubEngine$1.run(PubSubEngine.java:96) > > > at java.base@11.0.17 > > > > > > /java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) > > > at java.base@11.0.17 > > > /java.util.concurrent.FutureTask.run(FutureTask.java:264) > > > at java.base@11.0.17 > > > > > > > /java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) > > > at java.base@11.0.17 > > > > > > > /java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) > > > at java.base@11.0.17/java.lang.Thread.run(Thread.java:829) > > > > > > "socket_c2s_ssl-thread-2" #78 daemon prio=5 waiting on lock > > > java.lang.Thread.State: BLOCKED > > > - blocked on > > > org.jivesoftware.openfire.streammanagement.StreamManager@7e024771 > > (owned by > > > TaskEngine-pool-3 id=40) > > > at > > > > > > > org.jivesoftware.openfire.streammanagement.StreamManager.processClientAcknowledgement(StreamManager.java:493) > > > at > > > > > > > org.jivesoftware.openfire.streammanagement.StreamManager.process(StreamManager.java:185) > > > at > > > org.jivesoftware.openfire.net > > <http://org.jivesoftware.openfire.net > >.StanzaHandler.process(StanzaHandler.java:223) > > > at > > > > > > > org.jivesoftware.openfire.nio.ConnectionHandler.messageReceived(ConnectionHandler.java:178) > > > at > > > > > > > org.apache.mina.core.filterchain.DefaultIoFilterChain$TailFilter.messageReceived(DefaultIoFilterChain.java:1015) > > > at > > > > > > > org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:650) > > > at > > > > > > > org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1300(DefaultIoFilterChain.java:49) > > > at > > > > > > > org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.messageReceived(DefaultIoFilterChain.java:1128) > > > at > > > > > > > org.apache.mina.core.filterchain.IoFilterAdapter.messageReceived(IoFilterAdapter.java:122) > > > at > > > > > > > org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:650) > > > at > > > > > > > org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1300(DefaultIoFilterChain.java:49) > > > at > > > > > > > org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.messageReceived(DefaultIoFilterChain.java:1128) > > > at > > > > > > > org.apache.mina.filter.codec.AbstractProtocolDecoderOutput.flush(AbstractProtocolDecoderOutput.java:64) > > > at > > > > > > > org.apache.mina.filter.codec.ProtocolCodecFilter.messageReceived(ProtocolCodecFilter.java:249) > > > at > > > > > > > org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:650) > > > at > > > > > > > org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1300(DefaultIoFilterChain.java:49) > > > at > > > > > > > org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.messageReceived(DefaultIoFilterChain.java:1128) > > > at > > > > > > org.apache.mina.filter.ssl.SSLHandlerG0.receive_loop(SSLHandlerG0.java:236) > > > at > > > org.apache.mina.filter.ssl.SSLHandlerG0.receive(SSLHandlerG0.java:162) > > > - locked org.apache.mina.filter.ssl.SSLHandlerG0@99f49e2 > > > at > > > > > > org.apache.mina.filter.ssl.SslFilter.messageReceived(SslFilter.java:342) > > > at > > > > > > > org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:650) > > > at > > > > > > > org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1300(DefaultIoFilterChain.java:49) > > > at > > > > > > > org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.messageReceived(DefaultIoFilterChain.java:1128) > > > at > > > > > > org.apache.mina.core.filterchain.IoFilterEvent.fire(IoFilterEvent.java:106) > > > at org.apache.mina.core.session.IoEvent.run(IoEvent.java:89) > > > at > > > > > > > org.apache.mina.filter.executor.OrderedThreadPoolExecutor$Worker.runTask(OrderedThreadPoolExecutor.java:763) > > > at > > > > > > > org.apache.mina.filter.executor.OrderedThreadPoolExecutor$Worker.runTasks(OrderedThreadPoolExecutor.java:755) > > > at > > > > > > > org.apache.mina.filter.executor.OrderedThreadPoolExecutor$Worker.run(OrderedThreadPoolExecutor.java:695) > > > at java.base@11.0.17/java.lang.Thread.run(Thread.java:829) > > > > > > > -- > > *Emmanuel Lécharny - CTO* 205 Promenade des Anglais – 06200 NICE > > T. +33 (0)4 89 97 36 50 > > P. +33 (0)6 08 33 32 61 > > emmanuel.lecha...@busit.com <mailto:emmanuel.lecha...@busit.com> > > https://www.busit.com/ <https://www.busit.com/> > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: users-unsubscr...@mina.apache.org > > <mailto:users-unsubscr...@mina.apache.org> > > For additional commands, e-mail: users-h...@mina.apache.org > > <mailto:users-h...@mina.apache.org> > > > > -- > *Emmanuel Lécharny - CTO* 205 Promenade des Anglais – 06200 NICE > T. +33 (0)4 89 97 36 50 > P. +33 (0)6 08 33 32 61 > emmanuel.lecha...@busit.com https://www.busit.com/ >
