[
https://issues.apache.org/jira/browse/SSHD-844?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16623184#comment-16623184
]
Roberto Deandrea commented on SSHD-844:
---------------------------------------
Hi,
The workaround has nothing to do with Apache SSHD code, but relates to the way
our application proxies ftp-data client connections.
Kind Regards.
Roberto
> Deadlock detected in Apache SSHD client code
> --------------------------------------------
>
> Key: SSHD-844
> URL: https://issues.apache.org/jira/browse/SSHD-844
> Project: MINA SSHD
> Issue Type: Bug
> Affects Versions: 2.0.1
> Environment: AIX 7.2
> java version "1.8.0_151"
> Java(TM) SE Runtime Environment (build 8.0.5.7 -
> pap6480sr5fp7-20171216_01(SR5 FP7))
> IBM J9 VM (build 2.9, JRE 1.8.0 AIX ppc64-64 Compressed References
> 20171215_373586 (JIT enabled, AOT enabled)
> OpenJ9 - 5aa401f
> OMR - 101e793
> IBM - b4a79bf)
> JCL - 20171214_01 based on Oracle jdk8u151-b12
> Reporter: Roberto Deandrea
> Priority: Critical
> Attachments: javacore.20180913.153332.14221746.0001.txt, jca457.jar
>
>
> I found a deadlock in Apache SSHD client code running integration test for a
> local port forwarding application. Attached the IBM javacore with information
> about the deadlock.
> Please run $java -jar jca457.jar to load the javacore file and display thread
> stack traces.
> [^javacore.20180913.153332.14221746.0001.txt]
> [^jca457.jar]
>
> There are 2 threads in DEADLOCK belonging to client pool's threads, that
> blocked the other pool threads from working.
> The locks involved are LOCK1 (object ChannelOutputStream) and LOCK2 Object
> encodeLock of AbstractSession class.
>
> The below thread is waiting on LOCK2 and owns LOCK1 (that is
> ChannelOutputStream object)
> The thread sshd-SshClient[61e9f863]-nio2-thread-6 has the following stack
> trace :
>
> at
> org/apache/sshd/common/session/helpers/AbstractSession.doWritePacket(AbstractSession.java:1189(Compiled
> Code))
> at
> org/apache/sshd/common/session/helpers/AbstractSession.writePacket(AbstractSession.java:1137(Compiled
> Code))
> at
> org/apache/sshd/common/channel/AbstractChannel.writePacket(AbstractChannel.java:773(Compiled
> Code))
> at
> org/apache/sshd/common/channel/ChannelOutputStream.flush(ChannelOutputStream.java:227(Compiled
> Code))
> at
> org/apache/sshd/common/forward/DefaultForwardingFilter$StaticIoHandler.messageReceived(DefaultForwardingFilter.java:1063(Compiled
> Code))
> at
> org/apache/sshd/common/io/nio2/Nio2Session.handleReadCycleCompletion(Nio2Session.java:339(Compiled
> Code))
> at
> org/apache/sshd/common/io/nio2/Nio2Session$1.onCompleted(Nio2Session.java:318(Compiled
> Code))
> at
> org/apache/sshd/common/io/nio2/Nio2Session$1.onCompleted(Nio2Session.java:315(Compiled
> Code))
> at
> org/apache/sshd/common/io/nio2/Nio2CompletionHandler.lambda$completed$0(Nio2CompletionHandler.java:38(Compiled
> Code))
> at
> org/apache/sshd/common/io/nio2/Nio2CompletionHandler$$Lambda$254/0000000023F76250.run(Bytecode
> PC:12(Compiled Code))
> at
> java/security/AccessController.doPrivileged(AccessController.java:638(Compiled
> Code))
> at
> org/apache/sshd/common/io/nio2/Nio2CompletionHandler.completed(Nio2CompletionHandler.java:37(Compiled
> Code))
> at sun/nio/ch/Invoker.invokeUnchecked(Invoker.java:138(Compiled Code))
> at sun/nio/ch/Invoker$2.run(Invoker.java:230(Compiled Code))
> at
> sun/nio/ch/AsynchronousChannelGroupImpl$1.run(AsynchronousChannelGroupImpl.java:124(Compiled
> Code))
> at
> java/util/concurrent/ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1160(Compiled
> Code))
> at
> java/util/concurrent/ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
>
> at java/lang/Thread.run(Thread.java:811)
>
> This thread is waiting for LOCK1 and owns LOCK2.
> The thread sshd-SshClient[61e9f863]-nio2-thread-10 has the following stack
> trace:
>
> at
> org/apache/sshd/common/channel/ChannelOutputStream.close(ChannelOutputStream.java:249(Compiled
> Code))
> at
> org/apache/sshd/common/util/io/IoUtils.closeQuietly(IoUtils.java:151(Compiled
> Code))
> at
> org/apache/sshd/client/channel/AbstractClientChannel.lambda$getInnerCloseable$1(AbstractClientChannel.java:203(Compiled
> Code))
> at
> org/apache/sshd/client/channel/AbstractClientChannel$$Lambda$314/00000000252C5370.run(Bytecode
> PC:4(Compiled Code))
> at
> org/apache/sshd/common/util/closeable/Builder$1.doClose(Builder.java:47(Compiled
> Code))
> at
> org/apache/sshd/common/util/closeable/SimpleCloseable.close(SimpleCloseable.java:63(Compiled
> Code))
> at
> org/apache/sshd/common/util/closeable/SequentialCloseable$1.operationComplete(SequentialCloseable.java:56(Compiled
> Code))
> at
> org/apache/sshd/common/util/closeable/SequentialCloseable$1.operationComplete(SequentialCloseable.java:45(Compiled
> Code))
> at
> org/apache/sshd/common/future/AbstractSshFuture.notifyListener(AbstractSshFuture.java:159(Compiled
> Code))
> at
> org/apache/sshd/common/future/DefaultSshFuture.addListener(DefaultSshFuture.java:167(Compiled
> Code))
> at
> org/apache/sshd/common/util/closeable/SequentialCloseable$1.operationComplete(SequentialCloseable.java:57(Compiled
> Code))
> at
> org/apache/sshd/common/util/closeable/SequentialCloseable$1.operationComplete(SequentialCloseable.java:45(Compiled
> Code))
> at
> org/apache/sshd/common/util/closeable/SequentialCloseable.doClose(SequentialCloseable.java:69(Compiled
> Code))
> at
> org/apache/sshd/common/util/closeable/SimpleCloseable.close(SimpleCloseable.java:63(Compiled
> Code))
> at
> org/apache/sshd/common/util/closeable/SequentialCloseable$1.operationComplete(SequentialCloseable.java:56(Compiled
> Code))
> at
> org/apache/sshd/common/util/closeable/SequentialCloseable$1.operationComplete(SequentialCloseable.java:45(Compiled
> Code))
> at
> org/apache/sshd/common/future/AbstractSshFuture.notifyListener(AbstractSshFuture.java:159(Compiled
> Code))
> at
> org/apache/sshd/common/future/DefaultSshFuture.addListener(DefaultSshFuture.java:167(Compiled
> Code))
> at
> org/apache/sshd/common/util/closeable/SequentialCloseable$1.operationComplete(SequentialCloseable.java:57(Compiled
> Code))
> at
> org/apache/sshd/common/util/closeable/SequentialCloseable$1.operationComplete(SequentialCloseable.java:45(Compiled
> Code))
> at
> org/apache/sshd/common/util/closeable/SequentialCloseable.doClose(SequentialCloseable.java:69(Compiled
> Code))
> at
> org/apache/sshd/common/util/closeable/SimpleCloseable.close(SimpleCloseable.java:63(Compiled
> Code))
> at
> org/apache/sshd/common/util/closeable/AbstractInnerCloseable.doCloseGracefully(AbstractInnerCloseable.java:40(Compiled
> Code))
> at
> org/apache/sshd/common/util/closeable/AbstractCloseable.close(AbstractCloseable.java:98(Compiled
> Code))
> at
> org/apache/sshd/common/forward/DefaultForwardingFilter$StaticIoHandler.sessionClosed(DefaultForwardingFilter.java:1043(Compiled
> Code))
> at
> org/apache/sshd/common/io/nio2/Nio2Session.doCloseImmediately(Nio2Session.java:266(Compiled
> Code))
> at
> org/apache/sshd/common/util/closeable/AbstractCloseable.close(AbstractCloseable.java:83(Compiled
> Code))
> at
> org/apache/sshd/common/util/closeable/ParallelCloseable.doClose(ParallelCloseable.java:65(Compiled
> Code))
> at
> org/apache/sshd/common/util/closeable/SimpleCloseable.close(SimpleCloseable.java:63(Compiled
> Code))
> at
> org/apache/sshd/common/util/closeable/SequentialCloseable$1.operationComplete(SequentialCloseable.java:56(Compiled
> Code))
> at
> org/apache/sshd/common/util/closeable/SequentialCloseable$1.operationComplete(SequentialCloseable.java:45(Compiled
> Code))
> at
> org/apache/sshd/common/util/closeable/SequentialCloseable.doClose(SequentialCloseable.java:69(Compiled
> Code))
> at
> org/apache/sshd/common/util/closeable/SimpleCloseable.close(SimpleCloseable.java:63(Compiled
> Code))
> at
> org/apache/sshd/common/util/closeable/AbstractInnerCloseable.doCloseImmediately(AbstractInnerCloseable.java:46(Compiled
> Code))
> at
> org/apache/sshd/common/util/closeable/AbstractCloseable.close(AbstractCloseable.java:83(Compiled
> Code))
> at
> org/apache/sshd/common/util/closeable/SequentialCloseable$1.operationComplete(SequentialCloseable.java:56(Compiled
> Code))
> at
> org/apache/sshd/common/util/closeable/SequentialCloseable$1.operationComplete(SequentialCloseable.java:45(Compiled
> Code))
> at
> org/apache/sshd/common/future/AbstractSshFuture.notifyListener(AbstractSshFuture.java:159(Compiled
> Code))
> at
> org/apache/sshd/common/future/DefaultSshFuture.addListener(DefaultSshFuture.java:167(Compiled
> Code))
> at
> org/apache/sshd/common/util/closeable/SequentialCloseable$1.operationComplete(SequentialCloseable.java:57(Compiled
> Code))
> at
> org/apache/sshd/common/util/closeable/SequentialCloseable$1.operationComplete(SequentialCloseable.java:45(Compiled
> Code))
> at
> org/apache/sshd/common/util/closeable/SequentialCloseable.doClose(SequentialCloseable.java:69(Compiled
> Code))
> at
> org/apache/sshd/common/util/closeable/SimpleCloseable.close(SimpleCloseable.java:63(Compiled
> Code))
> at
> org/apache/sshd/common/util/closeable/AbstractInnerCloseable.doCloseImmediately(AbstractInnerCloseable.java:46(Compiled
> Code))
> at
> org/apache/sshd/common/util/closeable/AbstractCloseable.close(AbstractCloseable.java:83(Compiled
> Code))
> at
> org/apache/sshd/common/util/closeable/SequentialCloseable$1.operationComplete(SequentialCloseable.java:56(Compiled
> Code))
> at
> org/apache/sshd/common/util/closeable/SequentialCloseable$1.operationComplete(SequentialCloseable.java:45(Compiled
> Code))
> at
> org/apache/sshd/common/util/closeable/SequentialCloseable.doClose(SequentialCloseable.java:69(Compiled
> Code))
> at
> org/apache/sshd/common/util/closeable/SimpleCloseable.close(SimpleCloseable.java:63(Compiled
> Code))
> at
> org/apache/sshd/common/util/closeable/AbstractInnerCloseable.doCloseImmediately(AbstractInnerCloseable.java:46(Compiled
> Code))
> at
> org/apache/sshd/common/util/closeable/AbstractCloseable.close(AbstractCloseable.java:83(Compiled
> Code))
> at
> org/apache/sshd/common/util/closeable/ParallelCloseable.doClose(ParallelCloseable.java:65(Compiled
> Code))
> at
> org/apache/sshd/common/util/closeable/SimpleCloseable.close(SimpleCloseable.java:63(Compiled
> Code))
> at
> org/apache/sshd/common/util/closeable/SequentialCloseable$1.operationComplete(SequentialCloseable.java:56(Compiled
> Code))
> at
> org/apache/sshd/common/util/closeable/SequentialCloseable$1.operationComplete(SequentialCloseable.java:45(Compiled
> Code))
> at
> org/apache/sshd/common/util/closeable/SequentialCloseable.doClose(SequentialCloseable.java:69(Compiled
> Code))
> at
> org/apache/sshd/common/util/closeable/SimpleCloseable.close(SimpleCloseable.java:63(Compiled
> Code))
> at
> org/apache/sshd/common/util/closeable/AbstractInnerCloseable.doCloseImmediately(AbstractInnerCloseable.java:46(Compiled
> Code))
> at
> org/apache/sshd/common/util/closeable/AbstractCloseable.close(AbstractCloseable.java:83(Compiled
> Code))
> at
> org/apache/sshd/common/session/helpers/AbstractSession.exceptionCaught(AbstractSession.java:988)
>
> at
> org/apache/sshd/client/session/ClientSessionImpl.exceptionCaught(ClientSessionImpl.java:121)
>
> at
> org/apache/sshd/common/session/helpers/AbstractSessionIoHandler.exceptionCaught(AbstractSessionIoHandler.java:53)
>
> at
> org/apache/sshd/common/io/nio2/Nio2Session.exceptionCaught(Nio2Session.java:184(Compiled
> Code))
> at
> org/apache/sshd/common/io/nio2/Nio2Session.handleWriteCycleFailure(Nio2Session.java:460)
>
> at
> org/apache/sshd/common/io/nio2/Nio2Session$2.onFailed(Nio2Session.java:415)
> at
> org/apache/sshd/common/io/nio2/Nio2CompletionHandler.lambda$failed$1(Nio2CompletionHandler.java:46(Compiled
> Code))
> at
> org/apache/sshd/common/io/nio2/Nio2CompletionHandler$$Lambda$324/00000000252CCA50.run(Bytecode
> PC:12(Compiled Code))
> at
> java/security/AccessController.doPrivileged(AccessController.java:638(Compiled
> Code))
> at
> org/apache/sshd/common/io/nio2/Nio2CompletionHandler.failed(Nio2CompletionHandler.java:45(Compiled
> Code))
> at sun/nio/ch/Invoker.invokeUnchecked(Invoker.java:140(Compiled Code))
> at sun/nio/ch/Invoker.invokeDirect(Invoker.java:169(Compiled Code))
> at
> sun/nio/ch/UnixAsynchronousSocketChannelImpl.implWrite(UnixAsynchronousSocketChannelImpl.java:749(Compiled
> Code))
> at
> sun/nio/ch/AsynchronousSocketChannelImpl.write(AsynchronousSocketChannelImpl.java:394(Compiled
> Code))
> at
> sun/nio/ch/AsynchronousSocketChannelImpl.write(AsynchronousSocketChannelImpl.java:411(Compiled
> Code))
> at
> org/apache/sshd/common/io/nio2/Nio2Session.doWriteCycle(Nio2Session.java:401(Compiled
> Code))
> at
> org/apache/sshd/common/io/nio2/Nio2Session.startWriting(Nio2Session.java:386(Compiled
> Code))
> at
> org/apache/sshd/common/io/nio2/Nio2Session.writePacket(Nio2Session.java:167(Compiled
> Code))
> at
> org/apache/sshd/common/session/helpers/AbstractSession.doWritePacket(AbstractSession.java:1196(Compiled
> Code))
> at
> org/apache/sshd/common/session/helpers/AbstractSession.writePacket(AbstractSession.java:1137(Compiled
> Code))
> at
> org/apache/sshd/common/channel/AbstractChannel.writePacket(AbstractChannel.java:773(Compiled
> Code))
> at
> org/apache/sshd/common/channel/ChannelOutputStream.flush(ChannelOutputStream.java:227(Compiled
> Code))
> at
> org/apache/sshd/common/forward/DefaultForwardingFilter$StaticIoHandler.lambda$messageReceived$1(DefaultForwardingFilter.java:1069)
>
> at
> org/apache/sshd/common/forward/DefaultForwardingFilter$StaticIoHandler$$Lambda$327/00000000246D5DB0.operationComplete(Bytecode
> PC:20)
> at
> org/apache/sshd/common/future/AbstractSshFuture.notifyListener(AbstractSshFuture.java:159(Compiled
> Code))
> at
> org/apache/sshd/common/future/DefaultSshFuture.notifyListeners(DefaultSshFuture.java:217(Compiled
> Code))
> at
> org/apache/sshd/common/future/DefaultSshFuture.setValue(DefaultSshFuture.java:112(Compiled
> Code))
> at
> org/apache/sshd/client/future/DefaultOpenFuture.setOpened(DefaultOpenFuture.java:65(Compiled
> Code))
> at
> org/apache/sshd/client/channel/AbstractClientChannel.handleOpenSuccess(AbstractClientChannel.java:352(Compiled
> Code))
> at
> org/apache/sshd/common/session/helpers/AbstractConnectionService.channelOpenConfirmation(AbstractConnectionService.java:423(Compiled
> Code))
> at
> org/apache/sshd/common/session/helpers/AbstractConnectionService.process(AbstractConnectionService.java:333(Compiled
> Code))
> at
> org/apache/sshd/common/session/helpers/AbstractSession.doHandleMessage(AbstractSession.java:626(Compiled
> Code))
> at
> org/apache/sshd/common/session/helpers/AbstractSession.handleMessage(AbstractSession.java:559(Compiled
> Code))
> at
> org/apache/sshd/common/session/helpers/AbstractSession.decode(AbstractSession.java:1542(Compiled
> Code))
> at
> org/apache/sshd/common/session/helpers/AbstractSession.messageReceived(AbstractSession.java:520(Compiled
> Code))
> at
> org/apache/sshd/common/session/helpers/AbstractSessionIoHandler.messageReceived(AbstractSessionIoHandler.java:63(Compiled
> Code))
> at
> org/apache/sshd/common/io/nio2/Nio2Session.handleReadCycleCompletion(Nio2Session.java:339(Compiled
> Code))
> at
> org/apache/sshd/common/io/nio2/Nio2Session$1.onCompleted(Nio2Session.java:318(Compiled
> Code))
> at
> org/apache/sshd/common/io/nio2/Nio2Session$1.onCompleted(Nio2Session.java:315(Compiled
> Code))
> at
> org/apache/sshd/common/io/nio2/Nio2CompletionHandler.lambda$completed$0(Nio2CompletionHandler.java:38(Compiled
> Code))
> at
> org/apache/sshd/common/io/nio2/Nio2CompletionHandler$$Lambda$254/0000000023F76250.run(Bytecode
> PC:12(Compiled Code))
> at
> java/security/AccessController.doPrivileged(AccessController.java:638(Compiled
> Code))
> at
> org/apache/sshd/common/io/nio2/Nio2CompletionHandler.completed(Nio2CompletionHandler.java:37(Compiled
> Code))
> at sun/nio/ch/Invoker.invokeUnchecked(Invoker.java:138(Compiled Code))
> at sun/nio/ch/Invoker$2.run(Invoker.java:230(Compiled Code))
> at
> sun/nio/ch/AsynchronousChannelGroupImpl$1.run(AsynchronousChannelGroupImpl.java:124(Compiled
> Code))
> at
> java/util/concurrent/ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1160(Compiled
> Code))
> at
> java/util/concurrent/ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
>
> at java/lang/Thread.run(Thread.java:811)
>
> Can you find a solution to avoid this deadlock ?
> Thanks in advance
> Roberto
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
