Re: Deadlock when using SslFilter and ProxyFilter together
If you look at the stacktraces, I sent in my previous mails, you see, that the order, in which the filters are called, is right: When receiving: ... at org.apache.mina.filter.ssl.SslHandler.flushScheduledEvents(SslHandler.java:310) <--- SslFilter at org.apache.mina.filter.ssl.SslFilter.messageReceived(SslFilter.java:534) at org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:542) at org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1300(DefaultIoFilterChain.java:48) at org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.messageReceived(DefaultIoFilterChain.java:943) at org.apache.mina.proxy.filter.ProxyFilter.messageReceived(ProxyFilter.java:153) <--- ProxyFilter ... When sending: --- at org.apache.mina.proxy.filter.ProxyFilter.writeData(ProxyFilter.java:208) < ProxyFilter - waiting to lock <0x00076ef8b1d8> (a org.apache.mina.proxy.handlers.socks.Socks4LogicHandler) at org.apache.mina.proxy.filter.ProxyFilter.filterWrite(ProxyFilter.java:192) at org.apache.mina.core.filterchain.DefaultIoFilterChain.callPreviousFilterWrite(DefaultIoFilterChain.java:625) at org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1500(DefaultIoFilterChain.java:48) at org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.filterWrite(DefaultIoFilterChain.java:953) at org.apache.mina.filter.ssl.SslHandler.flushScheduledEvents(SslHandler.java:316) at org.apache.mina.filter.ssl.SslFilter.filterWrite(SslFilter.java:653) < SslFilter I am only writing the client part of an application. I communicate with a FIX/SSL server behind a SOCKS proxy, so I cannot change the order in which Ssl encryption and proxy framing must be done.
Re: Deadlock when using SslFilter and ProxyFilter together
Le 11/02/16 10:10, Norbert Irmer a écrit : > If you look at the stacktraces, I sent in my previous mails, you see, that > the order, in which the > filters are called, is right: > > When receiving: > > ... > at > org.apache.mina.filter.ssl.SslHandler.flushScheduledEvents(SslHandler.java:310) > <--- SslFilter > at org.apache.mina.filter.ssl.SslFilter.messageReceived(SslFilter.java:534) > at > org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:542) > at > org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1300(DefaultIoFilterChain.java:48) > at > org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.messageReceived(DefaultIoFilterChain.java:943) > at > org.apache.mina.proxy.filter.ProxyFilter.messageReceived(ProxyFilter.java:153) > > <--- ProxyFilter > ... > > > When sending: > > --- > at org.apache.mina.proxy.filter.ProxyFilter.writeData(ProxyFilter.java:208) >< ProxyFilter > - waiting to lock <0x00076ef8b1d8> (a > org.apache.mina.proxy.handlers.socks.Socks4LogicHandler) > at org.apache.mina.proxy.filter.ProxyFilter.filterWrite(ProxyFilter.java:192) > at > org.apache.mina.core.filterchain.DefaultIoFilterChain.callPreviousFilterWrite(DefaultIoFilterChain.java:625) > at > org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1500(DefaultIoFilterChain.java:48) > at > org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.filterWrite(DefaultIoFilterChain.java:953) > at > org.apache.mina.filter.ssl.SslHandler.flushScheduledEvents(SslHandler.java:316) > at org.apache.mina.filter.ssl.SslFilter.filterWrite(SslFilter.java:653) > < SslFilter > > > > I am only writing the client part of an application. I communicate with a > FIX/SSL server behind a SOCKS proxy, > so I cannot change the order in which Ssl encryption and proxy framing must > be done. Ok, I *think* that the problem is due to the fact that the flushScheduledEvents() method was not protecting writes *and* reads, so they can be executed at the same time by two threads. Then what happens is that both reach the ProxyFilter synchronized section, FixMessageProcessor takes the SslHandler lock FixMessageProcessor takes the ProxyFilter lock and at the same time : NioProcessor takes the ProxyFilter lock NioProcessor takes the SslHandler lock resulting in a Deadlock. NioProcessor >[ProxyFilter]--->[SslHandler]> IoHandler NioProcessor <[ProxyFilter]<---[SslHandler]< IoHandler The new version I have put on http://people.apache.org/~elecharny should solve the problem : instead of getting the lock whe we enter in the flushScheduledEvents() event, we exit immediately, doing nothing but incrementing the number of scheduled-events, instead of getting the lock. That means the thread which actually hold this lock will process all the enqueued messages itself, until the counter get down to 0. Can you give this version a try ?
Re: Deadlock when using SslFilter and ProxyFilter together
> The new version I have put on http://people.apache.org/~elecharny should > solve the problem : instead of getting the lock whe we enter in the > flushScheduledEvents() event, we exit immediately, doing nothing but > incrementing the number of scheduled-events, instead of getting the > lock. That means the thread which actually hold this lock will process > all the enqueued messages itself, until the counter get down to 0. > >Can you give this version a try ? Looks very good, I am running my application now for 20 minutes or so, and still have no deadlock. I am using now your snapshot versions without any changes. This is very nice, now it's possible to send a pull request for proxy support to the QuickFixJ people the next time you do a Mina release. Thanks a lot, Norbert
Re: Deadlock when using SslFilter and ProxyFilter together
Le 11/02/16 17:38, Norbert Irmer a écrit : > >> The new version I have put on http://people.apache.org/~elecharny should >> solve the problem : instead of getting the lock whe we enter in the >> flushScheduledEvents() event, we exit immediately, doing nothing but >> incrementing the number of scheduled-events, instead of getting the >> lock. That means the thread which actually hold this lock will process >> all the enqueued messages itself, until the counter get down to 0. >> >> Can you give this version a try ? > > Looks very good, I am running my application now for 20 minutes or so, and > still > have no deadlock. > > I am using now your snapshot versions without any changes. This is very nice, > now it's possible to send a pull request for proxy support to the QuickFixJ > people > the next time you do a Mina release. > > Thanks a lot, > Norbert > > Glad that seems to be fixed ! Norbert, I have to tell you that I really appreciated your patience, and I'm really sorry to have tried to deflect the prb for a couple of days. I really thought it was something else, and it took me some time (and the price was sleep deprivation :/) to realize that there were a deadlock. Worst part : this deadlock was already detected months ago by Terence Marks (https://issues.apache.org/jira/browse/DIRMINA-1019), a patch was proposed, and I fucked it up thinking it was overdoing. My bad... I'm going to cut a release asap. Expect it to be ready by the beginning of next week. thanks a lot !
Re: Deadlock when using SslFilter and ProxyFilter together
> seems like you have installed the proxy filter before the ssl filter in > your chain. Can you do the opposite ? I don't build the chain manually, I simply use a "ProxyConnector" (from package "org.apache.mina.proxy") to make a proxy connection (this also explains, why I had the SslFilter/autoStart problem in my earlier mail). "ProxyConnector" does the following in "setConnector": private void setConnector(final SocketConnector connector) { if (connector == null) { throw new IllegalArgumentException("connector cannot be null"); } this.connector = connector; String className = ProxyFilter.class.getName(); // Removes an old ProxyFilter instance from the chain if (connector.getFilterChain().contains(className)) { connector.getFilterChain().remove(className); } // Insert the ProxyFilter as the first filter in the filter chain builder connector.getFilterChain().addFirst(className, proxyFilter); }
Re: Deadlock when using SslFilter and ProxyFilter together
> Ok, but you can insert teh SslFilter *before* the proxyFilter. Once you > get the ProxyConnector instance, get the connector and do : Well, I am using a patched QuickFixJ version, in which the control flow is a bit different. Here is an excerpt from the code they are using: CompositeIoFilterChainBuilder ioFilterChainBuilder = new CompositeIoFilterChainBuilder( userIoFilterChainBuilder); boolean useProxy = proxyType != null && proxyPort > 0 && socketAddresses[0] instanceof InetSocketAddress; SslFilter sslFilter = null; if (sslEnabled) { boolean autoStart = !useProxy; sslFilter = installSslFilter(ioFilterChainBuilder, autoStart); } ioFilterChainBuilder.addLast(FIXProtocolCodecFactory.FILTER_NAME, new ProtocolCodecFilter(new FIXProtocolCodecFactory())); if (useProxy) { SocketConnector socketConnector = new NioSocketConnector(); socketConnector.setFilterChainBuilder(ioFilterChainBuilder); ioConnector = ProtocolFactory.createIoProxyConnector( socketConnector, (InetSocketAddress) socketAddresses[0], new InetSocketAddress(proxyHost, proxyPort), proxyType, proxyVersion, proxyUser, proxyPassword, proxyDomain, proxyWorkstation); } else { ioConnector = ProtocolFactory.createIoConnector(socketAddresses[0]); ioConnector.setFilterChainBuilder(ioFilterChainBuilder); } ioHandler = new InitiatorIoHandler(fixSession, sslFilter, networkingOptions, eventHandlingStrategy); } private SslFilter installSslFilter(CompositeIoFilterChainBuilder ioFilterChainBuilder, boolean autoStart) throws GeneralSecurityException { SslFilter sslFilter = new SslFilter(SSLContextFactory.getInstance(keyStoreName, keyStorePassword.toCharArray()), autoStart); if (enableProtocole != null) sslFilter.setEnabledProtocols(enableProtocole); if (cipherSuites != null) sslFilter.setEnabledCipherSuites(cipherSuites); sslFilter.setUseClientMode(true); ioFilterChainBuilder.addLast(SSLSupport.FILTER_NAME, sslFilter); return sslFilter; } Sure, with a lot of time, I could possibly rewrite the code, to swap the positions of the filters in the filterchain. But I see little sense in it, since the code is working. The only problem I have, is that I get this deadlock after a few minutes or so. At the moment, as a workaround, I simply disabled the "synchronized(handler) { ... }" in ProxyFilter.writeData(), I think for my special case this will work. I don't think it is possible, that the filterchains are traverted in the same order for reading and writing.
Re: Deadlock when using SslFilter and ProxyFilter together
Le 10/02/16 16:37, Norbert Irmer a écrit : >> seems like you have installed the proxy filter before the ssl filter in >> your chain. Can you do the opposite ? > I don't build the chain manually, I simply use a "ProxyConnector" (from > package "org.apache.mina.proxy") > to make a proxy connection (this also explains, why I had the > SslFilter/autoStart problem in my earlier mail). > > "ProxyConnector" does the following in "setConnector": > > > private void setConnector(final SocketConnector connector) { > if (connector == null) { > throw new IllegalArgumentException("connector cannot be null"); > } > > this.connector = connector; > String className = ProxyFilter.class.getName(); > > // Removes an old ProxyFilter instance from the chain > if (connector.getFilterChain().contains(className)) { > connector.getFilterChain().remove(className); > } > > // Insert the ProxyFilter as the first filter in the filter chain > builder > connector.getFilterChain().addFirst(className, proxyFilter); > } Ok, but you can insert teh SslFilter *before* the proxyFilter. Once you get the ProxyConnector instance, get the connector and do : SslFilter sslFilter = new SslFilter(BogusSslContextFactory .getInstance(true)); connector.getFilterChain().addFirst( "MySslFilter", sslFilter );
Re: Deadlock when using SslFilter and ProxyFilter together
Le 10/02/16 18:16, Norbert Irmer a écrit : >> Ok, but you can insert teh SslFilter *before* the proxyFilter. Once you >> get the ProxyConnector instance, get the connector and do : > > Well, I am using a patched QuickFixJ version, in which the control flow is a > bit different. > > Here is an excerpt from the code they are using: > > > CompositeIoFilterChainBuilder ioFilterChainBuilder = new > CompositeIoFilterChainBuilder( > userIoFilterChainBuilder); > > boolean useProxy = proxyType != null && proxyPort > 0 && > socketAddresses[0] instanceof > InetSocketAddress; > > > SslFilter sslFilter = null; > if (sslEnabled) { > boolean autoStart = !useProxy; > sslFilter = installSslFilter(ioFilterChainBuilder, autoStart); > } > > ioFilterChainBuilder.addLast(FIXProtocolCodecFactory.FILTER_NAME, > new ProtocolCodecFilter(new FIXProtocolCodecFactory())); > > if (useProxy) { > SocketConnector socketConnector = new NioSocketConnector(); > socketConnector.setFilterChainBuilder(ioFilterChainBuilder); > ioConnector = ProtocolFactory.createIoProxyConnector( > socketConnector, > (InetSocketAddress) socketAddresses[0], new > InetSocketAddress(proxyHost, > proxyPort), proxyType, proxyVersion, > proxyUser, proxyPassword, > proxyDomain, proxyWorkstation); > } else { > ioConnector = > ProtocolFactory.createIoConnector(socketAddresses[0]); > ioConnector.setFilterChainBuilder(ioFilterChainBuilder); > } > > ioHandler = new InitiatorIoHandler(fixSession, sslFilter, > networkingOptions, eventHandlingStrategy); > } > > private SslFilter installSslFilter(CompositeIoFilterChainBuilder > ioFilterChainBuilder, >boolean autoStart) > throws GeneralSecurityException { > SslFilter sslFilter = new > SslFilter(SSLContextFactory.getInstance(keyStoreName, > keyStorePassword.toCharArray()), autoStart); > if (enableProtocole != null) > sslFilter.setEnabledProtocols(enableProtocole); > if (cipherSuites != null) > sslFilter.setEnabledCipherSuites(cipherSuites); > sslFilter.setUseClientMode(true); > ioFilterChainBuilder.addLast(SSLSupport.FILTER_NAME, sslFilter); > return sslFilter; > } > > > Sure, with a lot of time, I could possibly rewrite the code, to swap the > positions of the filters in the filterchain. > > But I see little sense in it, since the code is working. The only problem I > have, is that I get this deadlock after a few minutes or so. How can you say the code is working when you get a deadlock ? As Jon said, having the ProcyFilter *before* the SSLFilter in the chain is not the right thing. The reason being that you may perfectly get fragmented TCP buffers, and the proxy might not be able to deal with de-fragmentation when it comes to encrypted data. I suggest that you patch teh code to first call the installSslFilter method after the proxy has been created, and in the installSslFiltermethod, use addFirst() instead of addLast(). Otherwise, you can also check with the latest trunk revision I built today to see if it's any better (on http://people.apache.org/~elecharny)
Re: Deadlock when using SslFilter and ProxyFilter together
I would have thought the reason SSL Filter needs to be before Proxy is because the SSL handshaking from the client side would get proxied away from the SSL Filter. On Wed, Feb 10, 2016 at 12:57 PM, Emmanuel Lécharnywrote: > Le 10/02/16 18:16, Norbert Irmer a écrit : > >> Ok, but you can insert teh SslFilter *before* the proxyFilter. Once you > >> get the ProxyConnector instance, get the connector and do : > > > > Well, I am using a patched QuickFixJ version, in which the control flow > is a bit different. > > > > Here is an excerpt from the code they are using: > > > > > > CompositeIoFilterChainBuilder ioFilterChainBuilder = new > CompositeIoFilterChainBuilder( > > userIoFilterChainBuilder); > > > > boolean useProxy = proxyType != null && proxyPort > 0 && > > socketAddresses[0] instanceof > InetSocketAddress; > > > > > > SslFilter sslFilter = null; > > if (sslEnabled) { > > boolean autoStart = !useProxy; > > sslFilter = installSslFilter(ioFilterChainBuilder, > autoStart); > > } > > > > > ioFilterChainBuilder.addLast(FIXProtocolCodecFactory.FILTER_NAME, > > new ProtocolCodecFilter(new > FIXProtocolCodecFactory())); > > > > if (useProxy) { > > SocketConnector socketConnector = new > NioSocketConnector(); > > > socketConnector.setFilterChainBuilder(ioFilterChainBuilder); > > ioConnector = ProtocolFactory.createIoProxyConnector( > socketConnector, > > (InetSocketAddress) socketAddresses[0], new > InetSocketAddress(proxyHost, > > proxyPort), proxyType, proxyVersion, > proxyUser, proxyPassword, > > proxyDomain, proxyWorkstation); > > } else { > > ioConnector = > ProtocolFactory.createIoConnector(socketAddresses[0]); > > ioConnector.setFilterChainBuilder(ioFilterChainBuilder); > > } > > > > ioHandler = new InitiatorIoHandler(fixSession, sslFilter, > networkingOptions, eventHandlingStrategy); > > } > > > > private SslFilter installSslFilter(CompositeIoFilterChainBuilder > ioFilterChainBuilder, > >boolean autoStart) > > throws GeneralSecurityException { > > SslFilter sslFilter = new > SslFilter(SSLContextFactory.getInstance(keyStoreName, > > keyStorePassword.toCharArray()), autoStart); > > if (enableProtocole != null) > > sslFilter.setEnabledProtocols(enableProtocole); > > if (cipherSuites != null) > > sslFilter.setEnabledCipherSuites(cipherSuites); > > sslFilter.setUseClientMode(true); > > ioFilterChainBuilder.addLast(SSLSupport.FILTER_NAME, > sslFilter); > > return sslFilter; > > } > > > > > > Sure, with a lot of time, I could possibly rewrite the code, to swap the > positions of the filters in the filterchain. > > > > But I see little sense in it, since the code is working. The only > problem I have, is that I get this deadlock after a few minutes or so. > How can you say the code is working when you get a deadlock ? > > As Jon said, having the ProcyFilter *before* the SSLFilter in the chain > is not the right thing. The reason being that you may perfectly get > fragmented TCP buffers, and the proxy might not be able to deal with > de-fragmentation when it comes to encrypted data. > > I suggest that you patch teh code to first call the installSslFilter > method after the proxy has been created, and in the > installSslFiltermethod, use addFirst() instead of addLast(). > > Otherwise, you can also check with the latest trunk revision I built > today to see if it's any better (on http://people.apache.org/~elecharny) > >
Re: Deadlock when using SslFilter and ProxyFilter together
Le 10/02/16 10:08, Norbert Irmer a écrit : > I attached a stacktrace of the two threads, which are in a deadlock. > > Jon maybe right, when saying the write stack is threadsafe, the problem is, > that simultaneously > a read operation is going on, and both stacks use the same filters. The stakcs you provided are useless. What we really need here is the full stack dump. Also note that the filter is instanciated as many times as there are sessions, so the real problem is when a session is processed by two different threads. That might make sense in some use case, but we have to guarantee that the encryption and decryption of messages are done fully before dealing with the next message. This is the reason we have a lock in the SslHandler. That being said, there *is* something wrong and we need to figure out what it is and how to get it fixed.
Re: Deadlock when using SslFilter and ProxyFilter together
I attached a stacktrace of the two threads, which are in a deadlock. Jon maybe right, when saying the write stack is threadsafe, the problem is, that simultaneously a read operation is going on, and both stacks use the same filters. Thread "NioProcessor-2": flushScheduledEvents():308, SslHandler {org.apache.mina.filter.ssl} messageReceived():534, SslFilter {org.apache.mina.filter.ssl} callNextMessageReceived():542, DefaultIoFilterChain {org.apache.mina.core.filterchain} access$1300():48, DefaultIoFilterChain {org.apache.mina.core.filterchain} messageReceived():943, DefaultIoFilterChain$EntryImpl$1 {org.apache.mina.core.filterchain} messageReceived():153, ProxyFilter {org.apache.mina.proxy.filter} callNextMessageReceived():542, DefaultIoFilterChain {org.apache.mina.core.filterchain} access$1300():48, DefaultIoFilterChain {org.apache.mina.core.filterchain} messageReceived():943, DefaultIoFilterChain$EntryImpl$1 {org.apache.mina.core.filterchain} messageReceived():109, IoFilterAdapter {org.apache.mina.core.filterchain} callNextMessageReceived():542, DefaultIoFilterChain {org.apache.mina.core.filterchain} fireMessageReceived():535, DefaultIoFilterChain {org.apache.mina.core.filterchain} read():684, AbstractPollingIoProcessor {org.apache.mina.core.polling} process():638, AbstractPollingIoProcessor {org.apache.mina.core.polling} process():627, AbstractPollingIoProcessor {org.apache.mina.core.polling} access$600():68, AbstractPollingIoProcessor {org.apache.mina.core.polling} run():1096, AbstractPollingIoProcessor$Processor {org.apache.mina.core.polling} run():64, NamePreservingRunnable {org.apache.mina.util} runWorker():1142, ThreadPoolExecutor {java.util.concurrent} run():617, ThreadPoolExecutor$Worker {java.util.concurrent} run():745, Thread {java.lang} Thread "FixMessageProcessor(0)::ABFX": writeData():209, ProxyFilter {org.apache.mina.proxy.filter} filterWrite():192, ProxyFilter {org.apache.mina.proxy.filter} callPreviousFilterWrite():625, DefaultIoFilterChain {org.apache.mina.core.filterchain} access$1500():48, DefaultIoFilterChain {org.apache.mina.core.filterchain} filterWrite():953, DefaultIoFilterChain$EntryImpl$1 {org.apache.mina.core.filterchain} flushScheduledEvents():316, SslHandler {org.apache.mina.filter.ssl} filterWrite():653, SslFilter {org.apache.mina.filter.ssl} callPreviousFilterWrite():625, DefaultIoFilterChain {org.apache.mina.core.filterchain} access$1500():48, DefaultIoFilterChain {org.apache.mina.core.filterchain} filterWrite():953, DefaultIoFilterChain$EntryImpl$1 {org.apache.mina.core.filterchain} filterWrite():321, ProtocolCodecFilter {org.apache.mina.filter.codec} callPreviousFilterWrite():625, DefaultIoFilterChain {org.apache.mina.core.filterchain} access$1500():48, DefaultIoFilterChain {org.apache.mina.core.filterchain} filterWrite():953, DefaultIoFilterChain$EntryImpl$1 {org.apache.mina.core.filterchain} filterWrite():877, DefaultIoFilterChain$TailFilter {org.apache.mina.core.filterchain} callPreviousFilterWrite():625, DefaultIoFilterChain {org.apache.mina.core.filterchain} fireFilterWrite():618, DefaultIoFilterChain {org.apache.mina.core.filterchain} write():530, AbstractIoSession {org.apache.mina.core.session} write():474, AbstractIoSession {org.apache.mina.core.session} send():51, IoSessionResponder {quickfix.mina} send():2534, Session {quickfix} sendRaw():2466, Session {quickfix} send():2519, Session {quickfix} sendToTarget():635, Session {quickfix} sendMessageToTarget():529, FixApplication {de.digitec.d3.gateway.fix} subscribe():52, FixProducerHandle {de.digitec.d3.gateway.fix} subscribe():134, FixApplication {de.digitec.d3.gateway.fix} processMessage():288, FixMessageProcessor {de.digitec.d3.gateway.fix} run():245, FixMessageProcessor {de.digitec.d3.gateway.fix} run():878, FixApplication$MessageProcessorThread {de.digitec.d3.gateway.fix} DIGITEC GmbH Johannes-Brahms-Platz 1 20355 Hamburg Tel +49 40 23776-147 Fax +49 40 23776-200 www.DIGITEC.de ___ DIGITEC GmbH EDV-Beratung, Soft- und Hardware Sitz Hamburg - HRB 288 18 Amtsgericht Hamburg Geschäftsführer: Hans-Joachim Hänel, Hans-Jürgen Joost Finanzamt Hamburg-Mitte, Steuer-Nr. 48/715/01586 - VAT-Nr. DE 118646977 WICHTIGER HINWEIS Die Kommunikation mit DIGITEC GmbH per Email dient nur dem Zweck der schnellen Information. Rechtsverbindliche Absprachen kommen über dieses Medium nicht zustande ! IMPORTANT NOTICE Communication with digitec GmbH via email is for quick information purposes only. This medium is not to be used for the exchange of legally binding statements !
Re: Deadlock when using SslFilter and ProxyFilter together
Hello, I did a "threaddump" with visualvm in the deadlock situation (not sure if this is what you wanted, but I don't see anything else), it shows the same deadlock situation (at the bottom of this mail), as in my previous mail. 2016-02-10 13:04:58 Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.60-b23 mixed mode): "Thread-7" #29 daemon prio=5 os_prio=0 tid=0x1ed49000 nid=0xfb0 runnable [0x1f66e000] java.lang.Thread.State: RUNNABLE at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.socketRead(SocketInputStream.java:116) at java.net.SocketInputStream.read(SocketInputStream.java:170) at java.net.SocketInputStream.read(SocketInputStream.java:141) at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284) at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326) at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178) - locked <0x00076ee06830> (a java.io.InputStreamReader) at java.io.InputStreamReader.read(InputStreamReader.java:184) at java.io.BufferedReader.fill(BufferedReader.java:161) at java.io.BufferedReader.skip(BufferedReader.java:411) - locked <0x00076ee06830> (a java.io.InputStreamReader) at de.digitec.d3.pricing.streaming.impl.StreamingChannelImpl.readMessage(StreamingChannelImpl.java:331) at de.digitec.d3.pricing.streaming.impl.Session.run(Session.java:116) - locked <0x00076ee068c8> (a java.lang.Object) Locked ownable synchronizers: - None "SiDispatcher_ReadLoop" #28 prio=5 os_prio=0 tid=0x1ed48800 nid=0x17fc waiting on condition [0x221de000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00076f04c960> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at de.digitec.d3.pricing.streaming.impl.SessionManagerAbstractImpl.readMessage(SessionManagerAbstractImpl.java:49) at de.digitec.d3.pricing.streaming.impl.SiDispatcher.run(SiDispatcher.java:455) at java.lang.Thread.run(Thread.java:745) Locked ownable synchronizers: - None "Thread-6" #27 daemon prio=5 os_prio=0 tid=0x1ed47800 nid=0x1674 runnable [0x2202f000] java.lang.Thread.State: RUNNABLE at java.net.DualStackPlainSocketImpl.accept0(Native Method) at java.net.DualStackPlainSocketImpl.socketAccept(DualStackPlainSocketImpl.java:131) at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:409) at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:199) - locked <0x00076ee102c8> (a java.net.SocksSocketImpl) at java.net.ServerSocket.implAccept(ServerSocket.java:545) at java.net.ServerSocket.accept(ServerSocket.java:513) at de.digitec.d3.pricing.streaming.impl.SocketAcceptor.accept(SocketAcceptor.java:152) at de.digitec.d3.pricing.streaming.impl.SocketAcceptor.run(SocketAcceptor.java:128) Locked ownable synchronizers: - None "SessionManagerStreamingImpl_Timer" #26 daemon prio=5 os_prio=0 tid=0x1ed47000 nid=0x1bbc in Object.wait() [0x21d9f000] java.lang.Thread.State: TIMED_WAITING (on object monitor) at java.lang.Object.wait(Native Method) at java.util.TimerThread.mainLoop(Timer.java:552) - locked <0x00076ee18180> (a java.util.TaskQueue) at java.util.TimerThread.run(Timer.java:505) Locked ownable synchronizers: - None "SiDispatcher_Timer" #25 prio=5 os_prio=0 tid=0x1ed46000 nid=0x12d8 in Object.wait() [0x2030f000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:502) at java.util.TimerThread.mainLoop(Timer.java:526) - locked <0x00076ee18350> (a java.util.TaskQueue) at java.util.TimerThread.run(Timer.java:505) Locked ownable synchronizers: - None "RMI TCP Connection(2)-172.16.125.86" #24 daemon prio=5 os_prio=0 tid=0x1ed45800 nid=0x1704 runnable [0x218ad000] java.lang.Thread.State: RUNNABLE at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.socketRead(SocketInputStream.java:116) at java.net.SocketInputStream.read(SocketInputStream.java:170) at java.net.SocketInputStream.read(SocketInputStream.java:141) at java.io.BufferedInputStream.fill(BufferedInputStream.java:246) at java.io.BufferedInputStream.read(BufferedInputStream.java:265) - locked <0x00076ee12618> (a java.io.BufferedInputStream) at
Re: Deadlock when using SslFilter and ProxyFilter together
Le 10/02/16 13:11, Norbert Irmer a écrit : > Hello, > > I did a "threaddump" with visualvm in the deadlock situation (not sure if > this is what you wanted, but I don't see > anything else), it shows the same deadlock situation (at the bottom of this > mail), as in my previous mail. Thanks ! seems like you have installed the proxy filter before the ssl filter in your chain. Can you do the opposite ?
Re: Deadlock when using SslFilter and ProxyFilter together
Le 10/02/16 22:11, Jon V. a écrit : > I would have thought the reason SSL Filter needs to be before Proxy is > because the SSL handshaking from the client side would get proxied away > from the SSL Filter. That too. But all in all, the handshake is just an exchange of data, as any other data beinge exchanged later on. It can also be split in pieces, the wame way...
Re: Deadlock when using SslFilter and ProxyFilter together
Obviously something is wrong with the thread-safe operation the write stack. If it was thread safe then there is no need to ever have locks in the filter chain unless you have an executor filter. 1027 exposes a bug in the lock routines of IoSession Sent from my iPhone > On Feb 9, 2016, at 12:48 PM, Emmanuel Lécharnywrote: > > Le 09/02/16 18:22, Jon V. a écrit : >> Its my understanding that Read and Write operations are thread-safe and >> should never cause deadlocks. (not sure why there are locks in those >> filters) >> >> Also, I don’t think SSL should be after Proxy. You cannot initiate an SSL >> session if you proxy the setup routine. What is the goal in this setup? >> >> Please send us the deadlock thread trace.(You can get it from VisualVM) > > I think this is a pb that is already exposed in DIRMINA-1027. > > I still have to find a couple of hours to analyse the code and find a > solution. >
Re: Deadlock when using SslFilter and ProxyFilter together
Its my understanding that Read and Write operations are thread-safe and should never cause deadlocks. (not sure why there are locks in those filters) Also, I don’t think SSL should be after Proxy. You cannot initiate an SSL session if you proxy the setup routine. What is the goal in this setup? Please send us the deadlock thread trace.(You can get it from VisualVM) On Tue, Feb 9, 2016 at 11:59 AM, Norbert Irmerwrote: > Hello, > > I detected a deadlock when using Proxies and Ssl together. > > The SslFilter and the ProxyFilzter both have synchronized blocks, and when > sending > and receiving messages you get deadlocks. > > The SslFilter hangs in "sslLock.lock()" in > "SslHandler.flushScheduledEvents()", > while the ProxyFilter hangs in "synchronized(handler)" in > "ProxyFilter.writeData". > > That sending and receiving messages with both filters simultaneously leads > to a deadlock > is clear, since both filters have locks and data flow direction is > reversed when reading and writing: > > Read direction: Socket --> ProxyFilter ---> SslFilter --> > FixProtocol --> Application > Write direction: Socket <-- ProxyFilter <--- SslFilter <-- > FixProtocol <-- Application > > > >
Re: Deadlock when using SslFilter and ProxyFilter together
Le 09/02/16 18:22, Jon V. a écrit : > Its my understanding that Read and Write operations are thread-safe and > should never cause deadlocks. (not sure why there are locks in those > filters) > > Also, I don’t think SSL should be after Proxy. You cannot initiate an SSL > session if you proxy the setup routine. What is the goal in this setup? > > Please send us the deadlock thread trace.(You can get it from VisualVM) I think this is a pb that is already exposed in DIRMINA-1027. I still have to find a couple of hours to analyse the code and find a solution.