[
https://issues.apache.org/jira/browse/SSHD-557?focusedWorklogId=437398&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-437398
]
ASF GitHub Bot logged work on SSHD-557:
---------------------------------------
Author: ASF GitHub Bot
Created on: 27/May/20 03:44
Start Date: 27/May/20 03:44
Worklog Time Spent: 10m
Work Description: gnodet commented on pull request #12:
URL: https://github.com/apache/mina-sshd/pull/12#issuecomment-633930238
Raised https://issues.apache.org/jira/browse/SSHD-1003
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 437398)
Remaining Estimate: 0h
Time Spent: 10m
> MINA SSHD deadlocks upon multiple concurrent HTTP downloads over
> locally-forwarded ports
> ----------------------------------------------------------------------------------------
>
> Key: SSHD-557
> URL: https://issues.apache.org/jira/browse/SSHD-557
> Project: MINA SSHD
> Issue Type: Bug
> Affects Versions: 0.14.0, 1.0.0
> Environment: Fedora 21, kernel 3.17.4-301, OpenJDK 1.8.0_51-b16
> Reporter: Sam Gardner
> Assignee: Lyor Goldstein
> Priority: Major
> Labels: sshd
> Fix For: 1.1.0
>
> Attachments: AbstractUserAuth.java, setup-tunnels-remote.sh,
> setup-tunnels.sh, ssh-load-remote.sh, ssh-load.sh,
> sshd-core-0.14.0-jar-with-dependencies.jar,
> sshd-core-1.0.0-jar-with-dependencies.jar
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> SSHD-Core 1.0.0 deadlocks when multiple concurrent downloads are in progress
> over ChannelOutputStreams
> In both Mina-SSHD 0.14.0 and 1.0.0 the server will become deadlocked if
> multiple concurrent downloads are started over a locally-forwarded TCP
> socket. In this state the process is still alive and client connections stay
> open, but no data can be transmitted and new connections will not be accepted
> - when inspected via telnet, the SSH port will not even return a banner.
> Inspection of the hung process using the {{jstack}} tool shows that each NIO
> worker thread is in the following state: {noformat}
> "sshd-SshServer[36f6e879]-nio2-thread-4" #14 daemon prio=5 os_prio=0
> tid=0x00007f219c009000 nid=0x34c6 in Object.wait() [0x00007f21c8439000]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0x00000000d799f9b0> (a
> org.apache.sshd.common.channel.Window)
> at java.lang.Object.wait(Object.java:502)
> at org.apache.sshd.common.channel.Window.waitForSpace(Window.java:175)
> - locked <0x00000000d799f9b0> (a
> org.apache.sshd.common.channel.Window)
> at
> org.apache.sshd.common.channel.ChannelOutputStream.flush(ChannelOutputStream.java:126)
> - locked <0x00000000d799fa68> (a
> org.apache.sshd.common.channel.ChannelOutputStream)
> at
> org.apache.sshd.server.forward.TcpipServerChannel$1.messageReceived(TcpipServerChannel.java:156)
> at
> org.apache.sshd.common.io.nio2.Nio2Session$2.onCompleted(Nio2Session.java:220)
> at
> org.apache.sshd.common.io.nio2.Nio2Session$2.onCompleted(Nio2Session.java:212)
> at
> org.apache.sshd.common.io.nio2.Nio2CompletionHandler$1.run(Nio2CompletionHandler.java:34)
> at java.security.AccessController.doPrivileged(Native Method)
> at
> org.apache.sshd.common.io.nio2.Nio2CompletionHandler.completed(Nio2CompletionHandler.java:31)
> at sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:126)
> at sun.nio.ch.Invoker.invokeDirect(Invoker.java:157)
> at
> sun.nio.ch.UnixAsynchronousSocketChannelImpl.implRead(UnixAsynchronousSocketChannelImpl.java:553)
> at
> sun.nio.ch.AsynchronousSocketChannelImpl.read(AsynchronousSocketChannelImpl.java:276)
> at
> sun.nio.ch.AsynchronousSocketChannelImpl.read(AsynchronousSocketChannelImpl.java:297)
> at
> java.nio.channels.AsynchronousSocketChannel.read(AsynchronousSocketChannel.java:420)
> at
> org.apache.sshd.common.io.nio2.Nio2Session.doReadCycle(Nio2Session.java:247)
> at
> org.apache.sshd.common.io.nio2.Nio2Session$2.onCompleted(Nio2Session.java:224)
> at
> org.apache.sshd.common.io.nio2.Nio2Session$2.onCompleted(Nio2Session.java:212)
> at
> org.apache.sshd.common.io.nio2.Nio2CompletionHandler$1.run(Nio2CompletionHandler.java:34)
> at java.security.AccessController.doPrivileged(Native Method)
> at
> org.apache.sshd.common.io.nio2.Nio2CompletionHandler.completed(Nio2CompletionHandler.java:31)
> at sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:126)
> at sun.nio.ch.Invoker.invokeDirect(Invoker.java:157)
> at
> sun.nio.ch.UnixAsynchronousSocketChannelImpl.implRead(UnixAsynchronousSocketChannelImpl.java:553)
> at
> sun.nio.ch.AsynchronousSocketChannelImpl.read(AsynchronousSocketChannelImpl.java:276)
> at
> sun.nio.ch.AsynchronousSocketChannelImpl.read(AsynchronousSocketChannelImpl.java:297)
> at
> java.nio.channels.AsynchronousSocketChannel.read(AsynchronousSocketChannel.java:420)
> at
> org.apache.sshd.common.io.nio2.Nio2Session.doReadCycle(Nio2Session.java:247)
> at
> org.apache.sshd.common.io.nio2.Nio2Session$2.onCompleted(Nio2Session.java:224)
> at
> org.apache.sshd.common.io.nio2.Nio2Session$2.onCompleted(Nio2Session.java:212)
> at
> org.apache.sshd.common.io.nio2.Nio2CompletionHandler$1.run(Nio2CompletionHandler.java:34)
> at java.security.AccessController.doPrivileged(Native Method)
> at
> org.apache.sshd.common.io.nio2.Nio2CompletionHandler.completed(Nio2CompletionHandler.java:31)
> at sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:126)
> at sun.nio.ch.Invoker.invokeDirect(Invoker.java:157)
> at
> sun.nio.ch.UnixAsynchronousSocketChannelImpl.implRead(UnixAsynchronousSocketChannelImpl.java:553)
> at
> sun.nio.ch.AsynchronousSocketChannelImpl.read(AsynchronousSocketChannelImpl.java:276)
> at
> sun.nio.ch.AsynchronousSocketChannelImpl.read(AsynchronousSocketChannelImpl.java:297)
> at
> java.nio.channels.AsynchronousSocketChannel.read(AsynchronousSocketChannel.java:420)
> at
> org.apache.sshd.common.io.nio2.Nio2Session.doReadCycle(Nio2Session.java:247)
> at
> org.apache.sshd.common.io.nio2.Nio2Session$2.onCompleted(Nio2Session.java:224)
> at
> org.apache.sshd.common.io.nio2.Nio2Session$2.onCompleted(Nio2Session.java:212)
> at
> org.apache.sshd.common.io.nio2.Nio2CompletionHandler$1.run(Nio2CompletionHandler.java:34)
> at java.security.AccessController.doPrivileged(Native Method)
> at
> org.apache.sshd.common.io.nio2.Nio2CompletionHandler.completed(Nio2CompletionHandler.java:31)
> at sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:126)
> at sun.nio.ch.Invoker.invokeDirect(Invoker.java:157)
> at
> sun.nio.ch.UnixAsynchronousSocketChannelImpl.implRead(UnixAsynchronousSocketChannelImpl.java:553)
> at
> sun.nio.ch.AsynchronousSocketChannelImpl.read(AsynchronousSocketChannelImpl.java:276)
> at
> sun.nio.ch.AsynchronousSocketChannelImpl.read(AsynchronousSocketChannelImpl.java:297)
> at
> java.nio.channels.AsynchronousSocketChannel.read(AsynchronousSocketChannel.java:420)
> at
> org.apache.sshd.common.io.nio2.Nio2Session.doReadCycle(Nio2Session.java:247)
> at
> org.apache.sshd.common.io.nio2.Nio2Session$2.onCompleted(Nio2Session.java:224)
> at
> org.apache.sshd.common.io.nio2.Nio2Session$2.onCompleted(Nio2Session.java:212)
> at
> org.apache.sshd.common.io.nio2.Nio2CompletionHandler$1.run(Nio2CompletionHandler.java:34)
> at java.security.AccessController.doPrivileged(Native Method)
> at
> org.apache.sshd.common.io.nio2.Nio2CompletionHandler.completed(Nio2CompletionHandler.java:31)
> at sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:126)
> at sun.nio.ch.Invoker.invokeDirect(Invoker.java:157)
> at
> sun.nio.ch.UnixAsynchronousSocketChannelImpl.implRead(UnixAsynchronousSocketChannelImpl.java:553)
> at
> sun.nio.ch.AsynchronousSocketChannelImpl.read(AsynchronousSocketChannelImpl.java:276)
> at
> sun.nio.ch.AsynchronousSocketChannelImpl.read(AsynchronousSocketChannelImpl.java:297)
> at
> java.nio.channels.AsynchronousSocketChannel.read(AsynchronousSocketChannel.java:420)
> at
> org.apache.sshd.common.io.nio2.Nio2Session.doReadCycle(Nio2Session.java:247)
> at
> org.apache.sshd.common.io.nio2.Nio2Session$2.onCompleted(Nio2Session.java:224)
> at
> org.apache.sshd.common.io.nio2.Nio2Session$2.onCompleted(Nio2Session.java:212)
> at
> org.apache.sshd.common.io.nio2.Nio2CompletionHandler$1.run(Nio2CompletionHandler.java:34)
> at java.security.AccessController.doPrivileged(Native Method)
> at
> org.apache.sshd.common.io.nio2.Nio2CompletionHandler.completed(Nio2CompletionHandler.java:31)
> at sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:126)
> at sun.nio.ch.Invoker.invokeDirect(Invoker.java:157)
> at
> sun.nio.ch.UnixAsynchronousSocketChannelImpl.implRead(UnixAsynchronousSocketChannelImpl.java:553)
> at
> sun.nio.ch.AsynchronousSocketChannelImpl.read(AsynchronousSocketChannelImpl.java:276)
> at
> sun.nio.ch.AsynchronousSocketChannelImpl.read(AsynchronousSocketChannelImpl.java:297)
> at
> java.nio.channels.AsynchronousSocketChannel.read(AsynchronousSocketChannel.java:420)
> at
> org.apache.sshd.common.io.nio2.Nio2Session.doReadCycle(Nio2Session.java:247)
> at
> org.apache.sshd.common.io.nio2.Nio2Session$2.onCompleted(Nio2Session.java:224)
> at
> org.apache.sshd.common.io.nio2.Nio2Session$2.onCompleted(Nio2Session.java:212)
> at
> org.apache.sshd.common.io.nio2.Nio2CompletionHandler$1.run(Nio2CompletionHandler.java:34)
> at java.security.AccessController.doPrivileged(Native Method)
> at
> org.apache.sshd.common.io.nio2.Nio2CompletionHandler.completed(Nio2CompletionHandler.java:31)
> at sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:126)
> at sun.nio.ch.Invoker.invokeDirect(Invoker.java:157)
> at
> sun.nio.ch.UnixAsynchronousSocketChannelImpl.implRead(UnixAsynchronousSocketChannelImpl.java:553)
> at
> sun.nio.ch.AsynchronousSocketChannelImpl.read(AsynchronousSocketChannelImpl.java:276)
> at
> sun.nio.ch.AsynchronousSocketChannelImpl.read(AsynchronousSocketChannelImpl.java:297)
> at
> java.nio.channels.AsynchronousSocketChannel.read(AsynchronousSocketChannel.java:420)
> at
> org.apache.sshd.common.io.nio2.Nio2Session.doReadCycle(Nio2Session.java:247)
> at
> org.apache.sshd.common.io.nio2.Nio2Session$2.onCompleted(Nio2Session.java:224)
> at
> org.apache.sshd.common.io.nio2.Nio2Session$2.onCompleted(Nio2Session.java:212)
> at
> org.apache.sshd.common.io.nio2.Nio2CompletionHandler$1.run(Nio2CompletionHandler.java:34)
> at java.security.AccessController.doPrivileged(Native Method)
> at
> org.apache.sshd.common.io.nio2.Nio2CompletionHandler.completed(Nio2CompletionHandler.java:31)
> at sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:126)
> at sun.nio.ch.Invoker.invokeDirect(Invoker.java:157)
> at
> sun.nio.ch.UnixAsynchronousSocketChannelImpl.implRead(UnixAsynchronousSocketChannelImpl.java:553)
> at
> sun.nio.ch.AsynchronousSocketChannelImpl.read(AsynchronousSocketChannelImpl.java:276)
> at
> sun.nio.ch.AsynchronousSocketChannelImpl.read(AsynchronousSocketChannelImpl.java:297)
> at
> java.nio.channels.AsynchronousSocketChannel.read(AsynchronousSocketChannel.java:420)
> at
> org.apache.sshd.common.io.nio2.Nio2Session.doReadCycle(Nio2Session.java:247)
> at
> org.apache.sshd.common.io.nio2.Nio2Session$2.onCompleted(Nio2Session.java:224)
> at
> org.apache.sshd.common.io.nio2.Nio2Session$2.onCompleted(Nio2Session.java:212)
> at
> org.apache.sshd.common.io.nio2.Nio2CompletionHandler$1.run(Nio2CompletionHandler.java:34)
> at java.security.AccessController.doPrivileged(Native Method)
> at
> org.apache.sshd.common.io.nio2.Nio2CompletionHandler.completed(Nio2CompletionHandler.java:31)
> at sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:126)
> at sun.nio.ch.Invoker.invokeDirect(Invoker.java:157)
> at
> sun.nio.ch.UnixAsynchronousSocketChannelImpl.implRead(UnixAsynchronousSocketChannelImpl.java:553)
> at
> sun.nio.ch.AsynchronousSocketChannelImpl.read(AsynchronousSocketChannelImpl.java:276)
> at
> sun.nio.ch.AsynchronousSocketChannelImpl.read(AsynchronousSocketChannelImpl.java:297)
> at
> java.nio.channels.AsynchronousSocketChannel.read(AsynchronousSocketChannel.java:420)
> at
> org.apache.sshd.common.io.nio2.Nio2Session.doReadCycle(Nio2Session.java:247)
> at
> org.apache.sshd.common.io.nio2.Nio2Session$2.onCompleted(Nio2Session.java:224)
> at
> org.apache.sshd.common.io.nio2.Nio2Session$2.onCompleted(Nio2Session.java:212)
> at
> org.apache.sshd.common.io.nio2.Nio2CompletionHandler$1.run(Nio2CompletionHandler.java:34)
> at java.security.AccessController.doPrivileged(Native Method)
> at
> org.apache.sshd.common.io.nio2.Nio2CompletionHandler.completed(Nio2CompletionHandler.java:31)
> at sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:126)
> at sun.nio.ch.Invoker$2.run(Invoker.java:218)
> at
> sun.nio.ch.AsynchronousChannelGroupImpl$1.run(AsynchronousChannelGroupImpl.java:112)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
> These are the steps to reproduce:
> # Start up approximately 200 separate SSH sessions with one local forward to
> an HTTP server:
> {{ssh -N -LXXXXX:<HTTP_SERVER>:80 account@<MINA_SSHD_SERVER>}}
> ## To simplify this process, I modified
> [org.apache.sshd.server.auth.AbstractUserAuth|^AbstractUserAuth.java] to
> always accept requests - this makes it possible to create the sessions on the
> same client (though I've also reproduced this on separate clients as well).
> ## This [script|^setup-tunnels.sh] will setup the client sessions as
> background processes
> # With the above sessions alive and backgrounded, fetch large files
> concurrently from the HTTP server over multiple forwarded ports:
> {{wget http://localhost:XXXXX/hugefile}}
> ## This [script|^ssh-load.sh] will start {{wget}} downloads in the above
> manner.
> # After a time, the {{wget}} downloads will stop progressing, and inspection
> of the SSH server process via {{jstack <PID>}} will show each of the
> {{nio2-thread-[0-9]}} threads in the deadlocked state.
> Note: I previously filed SSHD-539 on this, and have since reproduced it on
> vanilla MINA SSHD-Core without our proprietary modifications and running
> outside of Tomcat.
> The JAR files I used to reproduce are attached to this ticket as well.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
