subject:"\[jira\] \[Commented\] \(SSHD\-721\) deadlock\: all nio workers wait to be woken up"

[jira] [Commented] (SSHD-721) deadlock: all nio workers wait to be woken up

2018-05-21 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/SSHD-721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16482294#comment-16482294
 ] 

ASF GitHub Bot commented on SSHD-721:
-

Github user lgoldstein closed the pull request at:

https://github.com/apache/mina-sshd/pull/57


> deadlock: all nio workers wait to be woken up
> -
>
> Key: SSHD-721
> URL: https://issues.apache.org/jira/browse/SSHD-721
> Project: MINA SSHD
>  Issue Type: Bug
>Affects Versions: 1.3.0, 1.4.0
>Reporter: Markus Rathgeb
>Assignee: Guillaume Nodet
>Priority: Major
> Fix For: 2.0.0
>
>
> I am using sshd-core for a server machine (S) that accepts incoming 
> connections and port forwarding requests.
> There are client machines (C) that run servers that should be accessible by a 
> tunnel to the server.
> On the client machines (C) also an implementation using sshd-core is running 
> that establish the connection to the server (S) and initiate the port 
> forwarding.
> Other clients are using the tunnelled connection to communicate with the 
> servers that are running on the client machines (C).
> Sometimes I realized that no data is transferred anymore (through the 
> tunnels).
> All the worker reside in the waitFor function and no one wakes them up.
> {noformat}
> "sshd-SshServer[67de991c]-nio2-thread-3" - Thread t@125
>java.lang.Thread.State: TIMED_WAITING
>   at java.lang.Object.wait(Native Method)
>   - waiting on <132c6b60> (a java.lang.Object)
>   at 
> org.apache.sshd.client.channel.AbstractClientChannel.waitFor(AbstractClientChannel.java:244)
>   at 
> org.apache.sshd.common.forward.DefaultTcpipForwarder$StaticIoHandler.messageReceived(DefaultTcpipForwarder.java:984)
>   at 
> org.apache.sshd.common.io.nio2.Nio2Session.handleReadCycleCompletion(Nio2Session.java:276)
>   at 
> org.apache.sshd.common.io.nio2.Nio2Session$1.onCompleted(Nio2Session.java:256)
>   at 
> org.apache.sshd.common.io.nio2.Nio2Session$1.onCompleted(Nio2Session.java:253)
>   at 
> org.apache.sshd.common.io.nio2.Nio2CompletionHandler.lambda$completed$0(Nio2CompletionHandler.java:38)
>   at 
> org.apache.sshd.common.io.nio2.Nio2CompletionHandler$$Lambda$45/1071326492.run(Unknown
>  Source)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at 
> org.apache.sshd.common.io.nio2.Nio2CompletionHandler.completed(Nio2CompletionHandler.java:37)
>   at sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:126)
>   at sun.nio.ch.Invoker$2.run(Invoker.java:218)
>   at 
> sun.nio.ch.AsynchronousChannelGroupImpl$1.run(AsynchronousChannelGroupImpl.java:112)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>   at java.lang.Thread.run(Thread.java:745)
>Locked ownable synchronizers:
>   - locked <6d02d7ed> (a java.util.concurrent.ThreadPoolExecutor$Worker)
> "sshd-SshServer[67de991c]-nio2-thread-2" - Thread t@124
>java.lang.Thread.State: TIMED_WAITING
>   at java.lang.Object.wait(Native Method)
>   - waiting on <7e9f4eff> (a java.lang.Object)
>   at 
> org.apache.sshd.client.channel.AbstractClientChannel.waitFor(AbstractClientChannel.java:244)
>   at 
> org.apache.sshd.common.forward.DefaultTcpipForwarder$StaticIoHandler.messageReceived(DefaultTcpipForwarder.java:984)
>   at 
> org.apache.sshd.common.io.nio2.Nio2Session.handleReadCycleCompletion(Nio2Session.java:276)
>   at 
> org.apache.sshd.common.io.nio2.Nio2Session$1.onCompleted(Nio2Session.java:256)
>   at 
> org.apache.sshd.common.io.nio2.Nio2Session$1.onCompleted(Nio2Session.java:253)
>   at 
> org.apache.sshd.common.io.nio2.Nio2CompletionHandler.lambda$completed$0(Nio2CompletionHandler.java:38)
>   at 
> org.apache.sshd.common.io.nio2.Nio2CompletionHandler$$Lambda$45/1071326492.run(Unknown
>  Source)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at 
> org.apache.sshd.common.io.nio2.Nio2CompletionHandler.completed(Nio2CompletionHandler.java:37)
>   at sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:126)
>   at sun.nio.ch.Invoker$2.run(Invoker.java:218)
>   at 
> sun.nio.ch.AsynchronousChannelGroupImpl$1.run(AsynchronousChannelGroupImpl.java:112)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>   at java.lang.Thread.run(Thread.java:745)
>Locked ownable synchronizers:
>   - locked <35fbf3e8> (a java.util.concurrent.ThreadPoolExecutor$Worker)
> "sshd-SshServer[67de991c]-nio2-thread-1" - Thread t@122
>java.lang.Thread.State: TIMED_WAITING
>   at

[jira] [Commented] (SSHD-721) deadlock: all nio workers wait to be woken up

2018-05-06 Thread Goldstein Lyor (JIRA)


[ 
https://issues.apache.org/jira/browse/SSHD-721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16464994#comment-16464994
 ] 

Goldstein Lyor commented on SSHD-721:
-

[~bkuker1] see https://github.com/apache/mina-sshd/pull/57 - I think this is 
more along the lines that you mentioned. Please test it (if possible) and let 
us know how it goes.

> deadlock: all nio workers wait to be woken up
> -
>
> Key: SSHD-721
> URL: https://issues.apache.org/jira/browse/SSHD-721
> Project: MINA SSHD
>  Issue Type: Bug
>Affects Versions: 1.3.0, 1.4.0
>Reporter: Markus Rathgeb
>Assignee: Guillaume Nodet
>Priority: Major
> Fix For: 2.0.0
>
>
> I am using sshd-core for a server machine (S) that accepts incoming 
> connections and port forwarding requests.
> There are client machines (C) that run servers that should be accessible by a 
> tunnel to the server.
> On the client machines (C) also an implementation using sshd-core is running 
> that establish the connection to the server (S) and initiate the port 
> forwarding.
> Other clients are using the tunnelled connection to communicate with the 
> servers that are running on the client machines (C).
> Sometimes I realized that no data is transferred anymore (through the 
> tunnels).
> All the worker reside in the waitFor function and no one wakes them up.
> {noformat}
> "sshd-SshServer[67de991c]-nio2-thread-3" - Thread t@125
>java.lang.Thread.State: TIMED_WAITING
>   at java.lang.Object.wait(Native Method)
>   - waiting on <132c6b60> (a java.lang.Object)
>   at 
> org.apache.sshd.client.channel.AbstractClientChannel.waitFor(AbstractClientChannel.java:244)
>   at 
> org.apache.sshd.common.forward.DefaultTcpipForwarder$StaticIoHandler.messageReceived(DefaultTcpipForwarder.java:984)
>   at 
> org.apache.sshd.common.io.nio2.Nio2Session.handleReadCycleCompletion(Nio2Session.java:276)
>   at 
> org.apache.sshd.common.io.nio2.Nio2Session$1.onCompleted(Nio2Session.java:256)
>   at 
> org.apache.sshd.common.io.nio2.Nio2Session$1.onCompleted(Nio2Session.java:253)
>   at 
> org.apache.sshd.common.io.nio2.Nio2CompletionHandler.lambda$completed$0(Nio2CompletionHandler.java:38)
>   at 
> org.apache.sshd.common.io.nio2.Nio2CompletionHandler$$Lambda$45/1071326492.run(Unknown
>  Source)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at 
> org.apache.sshd.common.io.nio2.Nio2CompletionHandler.completed(Nio2CompletionHandler.java:37)
>   at sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:126)
>   at sun.nio.ch.Invoker$2.run(Invoker.java:218)
>   at 
> sun.nio.ch.AsynchronousChannelGroupImpl$1.run(AsynchronousChannelGroupImpl.java:112)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>   at java.lang.Thread.run(Thread.java:745)
>Locked ownable synchronizers:
>   - locked <6d02d7ed> (a java.util.concurrent.ThreadPoolExecutor$Worker)
> "sshd-SshServer[67de991c]-nio2-thread-2" - Thread t@124
>java.lang.Thread.State: TIMED_WAITING
>   at java.lang.Object.wait(Native Method)
>   - waiting on <7e9f4eff> (a java.lang.Object)
>   at 
> org.apache.sshd.client.channel.AbstractClientChannel.waitFor(AbstractClientChannel.java:244)
>   at 
> org.apache.sshd.common.forward.DefaultTcpipForwarder$StaticIoHandler.messageReceived(DefaultTcpipForwarder.java:984)
>   at 
> org.apache.sshd.common.io.nio2.Nio2Session.handleReadCycleCompletion(Nio2Session.java:276)
>   at 
> org.apache.sshd.common.io.nio2.Nio2Session$1.onCompleted(Nio2Session.java:256)
>   at 
> org.apache.sshd.common.io.nio2.Nio2Session$1.onCompleted(Nio2Session.java:253)
>   at 
> org.apache.sshd.common.io.nio2.Nio2CompletionHandler.lambda$completed$0(Nio2CompletionHandler.java:38)
>   at 
> org.apache.sshd.common.io.nio2.Nio2CompletionHandler$$Lambda$45/1071326492.run(Unknown
>  Source)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at 
> org.apache.sshd.common.io.nio2.Nio2CompletionHandler.completed(Nio2CompletionHandler.java:37)
>   at sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:126)
>   at sun.nio.ch.Invoker$2.run(Invoker.java:218)
>   at 
> sun.nio.ch.AsynchronousChannelGroupImpl$1.run(AsynchronousChannelGroupImpl.java:112)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>   at java.lang.Thread.run(Thread.java:745)
>Locked ownable synchronizers:
>   - locked <35fbf3e8> (a java.util.concurrent.ThreadPoolExecutor$Worker)
> "sshd-SshServer[67de991c]-nio2-thread-1" -

[jira] [Commented] (SSHD-721) deadlock: all nio workers wait to be woken up

2018-05-06 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/SSHD-721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16464993#comment-16464993
 ] 

ASF GitHub Bot commented on SSHD-721:
-

GitHub user lgoldstein opened a pull request:

https://github.com/apache/mina-sshd/pull/57

[SSHD-721] Close only SSH channel and not the session if error encountered 
during tunnel data read/write

See attached

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/lgoldstein/mina-sshd SSHD-721

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/mina-sshd/pull/57.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #57


commit c6fac3a56fcc813e182767aafe393247d06b6b76
Author: Goldstein Lyor 
Date:   2018-05-06T06:53:56Z

[SSHD-721] Close only SSH channel and not the session if error encountered 
during read/write of data through a forwarding tunnel




> deadlock: all nio workers wait to be woken up
> -
>
> Key: SSHD-721
> URL: https://issues.apache.org/jira/browse/SSHD-721
> Project: MINA SSHD
>  Issue Type: Bug
>Affects Versions: 1.3.0, 1.4.0
>Reporter: Markus Rathgeb
>Assignee: Guillaume Nodet
>Priority: Major
> Fix For: 2.0.0
>
>
> I am using sshd-core for a server machine (S) that accepts incoming 
> connections and port forwarding requests.
> There are client machines (C) that run servers that should be accessible by a 
> tunnel to the server.
> On the client machines (C) also an implementation using sshd-core is running 
> that establish the connection to the server (S) and initiate the port 
> forwarding.
> Other clients are using the tunnelled connection to communicate with the 
> servers that are running on the client machines (C).
> Sometimes I realized that no data is transferred anymore (through the 
> tunnels).
> All the worker reside in the waitFor function and no one wakes them up.
> {noformat}
> "sshd-SshServer[67de991c]-nio2-thread-3" - Thread t@125
>java.lang.Thread.State: TIMED_WAITING
>   at java.lang.Object.wait(Native Method)
>   - waiting on <132c6b60> (a java.lang.Object)
>   at 
> org.apache.sshd.client.channel.AbstractClientChannel.waitFor(AbstractClientChannel.java:244)
>   at 
> org.apache.sshd.common.forward.DefaultTcpipForwarder$StaticIoHandler.messageReceived(DefaultTcpipForwarder.java:984)
>   at 
> org.apache.sshd.common.io.nio2.Nio2Session.handleReadCycleCompletion(Nio2Session.java:276)
>   at 
> org.apache.sshd.common.io.nio2.Nio2Session$1.onCompleted(Nio2Session.java:256)
>   at 
> org.apache.sshd.common.io.nio2.Nio2Session$1.onCompleted(Nio2Session.java:253)
>   at 
> org.apache.sshd.common.io.nio2.Nio2CompletionHandler.lambda$completed$0(Nio2CompletionHandler.java:38)
>   at 
> org.apache.sshd.common.io.nio2.Nio2CompletionHandler$$Lambda$45/1071326492.run(Unknown
>  Source)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at 
> org.apache.sshd.common.io.nio2.Nio2CompletionHandler.completed(Nio2CompletionHandler.java:37)
>   at sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:126)
>   at sun.nio.ch.Invoker$2.run(Invoker.java:218)
>   at 
> sun.nio.ch.AsynchronousChannelGroupImpl$1.run(AsynchronousChannelGroupImpl.java:112)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>   at java.lang.Thread.run(Thread.java:745)
>Locked ownable synchronizers:
>   - locked <6d02d7ed> (a java.util.concurrent.ThreadPoolExecutor$Worker)
> "sshd-SshServer[67de991c]-nio2-thread-2" - Thread t@124
>java.lang.Thread.State: TIMED_WAITING
>   at java.lang.Object.wait(Native Method)
>   - waiting on <7e9f4eff> (a java.lang.Object)
>   at 
> org.apache.sshd.client.channel.AbstractClientChannel.waitFor(AbstractClientChannel.java:244)
>   at 
> org.apache.sshd.common.forward.DefaultTcpipForwarder$StaticIoHandler.messageReceived(DefaultTcpipForwarder.java:984)
>   at 
> org.apache.sshd.common.io.nio2.Nio2Session.handleReadCycleCompletion(Nio2Session.java:276)
>   at 
> org.apache.sshd.common.io.nio2.Nio2Session$1.onCompleted(Nio2Session.java:256)
>   at 
> org.apache.sshd.common.io.nio2.Nio2Session$1.onCompleted(Nio2Session.java:253)
>   at 
> org.apache.sshd.common.io.nio2.Nio2CompletionHandler.lambda$completed$0(Nio2CompletionHandler.java:38)
>   at 
> org.apache.sshd.common.io.nio2.Nio2CompletionHandler$$Lambda$45/1071326492.run(Unknown
>  Source)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at 
>

[jira] [Commented] (SSHD-721) deadlock: all nio workers wait to be woken up

2018-05-03 Thread Bill Kuker (JIRA)


[ 
https://issues.apache.org/jira/browse/SSHD-721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16462632#comment-16462632
 ] 

Bill Kuker commented on SSHD-721:
-

[~gnt] Thanks very much for your help, there is an issue with your patch. The 
call to {{channel.getSession().exceptionCaught(e);}} is incorrect.

It causes the connection between the sshd server and ssh client to be closed 
when an error occurs on the socket between the sshd server and the "web 
browser," or whatever the application making the connection through the port 
forward is.

I think there is a tough to make race condition, as that error has to occur on 
the first, or an early, run through the new code when the open future is not 
yet resolved. I am not familiar enough with the code to understand where the 
exception would have gone when thrown from the synchronous version.

I have 100% confirmed that end users are not able to reliably use real web 
browsers and real web servers under real world conditions *unless that line is 
removed*. I am just not sure where, if anywhere, that exception should go if it 
happens.

> deadlock: all nio workers wait to be woken up
> -
>
> Key: SSHD-721
> URL: https://issues.apache.org/jira/browse/SSHD-721
> Project: MINA SSHD
>  Issue Type: Bug
>Affects Versions: 1.3.0, 1.4.0
>Reporter: Markus Rathgeb
>Assignee: Guillaume Nodet
>Priority: Major
> Fix For: 2.0.0
>
>
> I am using sshd-core for a server machine (S) that accepts incoming 
> connections and port forwarding requests.
> There are client machines (C) that run servers that should be accessible by a 
> tunnel to the server.
> On the client machines (C) also an implementation using sshd-core is running 
> that establish the connection to the server (S) and initiate the port 
> forwarding.
> Other clients are using the tunnelled connection to communicate with the 
> servers that are running on the client machines (C).
> Sometimes I realized that no data is transferred anymore (through the 
> tunnels).
> All the worker reside in the waitFor function and no one wakes them up.
> {noformat}
> "sshd-SshServer[67de991c]-nio2-thread-3" - Thread t@125
>java.lang.Thread.State: TIMED_WAITING
>   at java.lang.Object.wait(Native Method)
>   - waiting on <132c6b60> (a java.lang.Object)
>   at 
> org.apache.sshd.client.channel.AbstractClientChannel.waitFor(AbstractClientChannel.java:244)
>   at 
> org.apache.sshd.common.forward.DefaultTcpipForwarder$StaticIoHandler.messageReceived(DefaultTcpipForwarder.java:984)
>   at 
> org.apache.sshd.common.io.nio2.Nio2Session.handleReadCycleCompletion(Nio2Session.java:276)
>   at 
> org.apache.sshd.common.io.nio2.Nio2Session$1.onCompleted(Nio2Session.java:256)
>   at 
> org.apache.sshd.common.io.nio2.Nio2Session$1.onCompleted(Nio2Session.java:253)
>   at 
> org.apache.sshd.common.io.nio2.Nio2CompletionHandler.lambda$completed$0(Nio2CompletionHandler.java:38)
>   at 
> org.apache.sshd.common.io.nio2.Nio2CompletionHandler$$Lambda$45/1071326492.run(Unknown
>  Source)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at 
> org.apache.sshd.common.io.nio2.Nio2CompletionHandler.completed(Nio2CompletionHandler.java:37)
>   at sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:126)
>   at sun.nio.ch.Invoker$2.run(Invoker.java:218)
>   at 
> sun.nio.ch.AsynchronousChannelGroupImpl$1.run(AsynchronousChannelGroupImpl.java:112)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>   at java.lang.Thread.run(Thread.java:745)
>Locked ownable synchronizers:
>   - locked <6d02d7ed> (a java.util.concurrent.ThreadPoolExecutor$Worker)
> "sshd-SshServer[67de991c]-nio2-thread-2" - Thread t@124
>java.lang.Thread.State: TIMED_WAITING
>   at java.lang.Object.wait(Native Method)
>   - waiting on <7e9f4eff> (a java.lang.Object)
>   at 
> org.apache.sshd.client.channel.AbstractClientChannel.waitFor(AbstractClientChannel.java:244)
>   at 
> org.apache.sshd.common.forward.DefaultTcpipForwarder$StaticIoHandler.messageReceived(DefaultTcpipForwarder.java:984)
>   at 
> org.apache.sshd.common.io.nio2.Nio2Session.handleReadCycleCompletion(Nio2Session.java:276)
>   at 
> org.apache.sshd.common.io.nio2.Nio2Session$1.onCompleted(Nio2Session.java:256)
>   at 
> org.apache.sshd.common.io.nio2.Nio2Session$1.onCompleted(Nio2Session.java:253)
>   at 
> org.apache.sshd.common.io.nio2.Nio2CompletionHandler.lambda$completed$0(Nio2CompletionHandler.java:38)
>   at 
> org.apache.sshd.common.io.nio2.Nio2CompletionHandler$$Lambda$45/1071326492.run(Unknown
>  Source)
>   at

[jira] [Commented] (SSHD-721) deadlock: all nio workers wait to be woken up

2018-05-02 Thread Bill Kuker (JIRA)


[ 
https://issues.apache.org/jira/browse/SSHD-721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16460874#comment-16460874
 ] 

Bill Kuker commented on SSHD-721:
-

Hello,

I am pretty certain now that the line
channel.getSession().exceptionCaught(e);
causes a problem. If an exception occurs on the socket between the port forward 
and the 'web browser' then the entire session between the ssh server and ssh 
client is closed, which is for sure not correct.

This is actually pretty common in the real world - when a user navigates to a 
new page while there are requests pending some browsers just close the 
connection abruptly. When the ssh server tries to write the response back to 
the browser it finds out that the connection is closed, calls exceptionCaught 
which ends up calling session.close().

Closing the entire ssh session on a sshd / ssh client communication error is 
correct, but closing the entire ssh session on an error on the incoming our 
outgoing port forward is wrong.

Not sure if I should re-open this bug or open a new one? Under ideal test 
conditions this bug is "fixed" but adding real world messiness the patch still 
needs a correction.

> deadlock: all nio workers wait to be woken up
> -
>
> Key: SSHD-721
> URL: https://issues.apache.org/jira/browse/SSHD-721
> Project: MINA SSHD
>  Issue Type: Bug
>Affects Versions: 1.3.0, 1.4.0
>Reporter: Markus Rathgeb
>Assignee: Guillaume Nodet
>Priority: Major
> Fix For: 2.0.0
>
>
> I am using sshd-core for a server machine (S) that accepts incoming 
> connections and port forwarding requests.
> There are client machines (C) that run servers that should be accessible by a 
> tunnel to the server.
> On the client machines (C) also an implementation using sshd-core is running 
> that establish the connection to the server (S) and initiate the port 
> forwarding.
> Other clients are using the tunnelled connection to communicate with the 
> servers that are running on the client machines (C).
> Sometimes I realized that no data is transferred anymore (through the 
> tunnels).
> All the worker reside in the waitFor function and no one wakes them up.
> {noformat}
> "sshd-SshServer[67de991c]-nio2-thread-3" - Thread t@125
>java.lang.Thread.State: TIMED_WAITING
>   at java.lang.Object.wait(Native Method)
>   - waiting on <132c6b60> (a java.lang.Object)
>   at 
> org.apache.sshd.client.channel.AbstractClientChannel.waitFor(AbstractClientChannel.java:244)
>   at 
> org.apache.sshd.common.forward.DefaultTcpipForwarder$StaticIoHandler.messageReceived(DefaultTcpipForwarder.java:984)
>   at 
> org.apache.sshd.common.io.nio2.Nio2Session.handleReadCycleCompletion(Nio2Session.java:276)
>   at 
> org.apache.sshd.common.io.nio2.Nio2Session$1.onCompleted(Nio2Session.java:256)
>   at 
> org.apache.sshd.common.io.nio2.Nio2Session$1.onCompleted(Nio2Session.java:253)
>   at 
> org.apache.sshd.common.io.nio2.Nio2CompletionHandler.lambda$completed$0(Nio2CompletionHandler.java:38)
>   at 
> org.apache.sshd.common.io.nio2.Nio2CompletionHandler$$Lambda$45/1071326492.run(Unknown
>  Source)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at 
> org.apache.sshd.common.io.nio2.Nio2CompletionHandler.completed(Nio2CompletionHandler.java:37)
>   at sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:126)
>   at sun.nio.ch.Invoker$2.run(Invoker.java:218)
>   at 
> sun.nio.ch.AsynchronousChannelGroupImpl$1.run(AsynchronousChannelGroupImpl.java:112)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>   at java.lang.Thread.run(Thread.java:745)
>Locked ownable synchronizers:
>   - locked <6d02d7ed> (a java.util.concurrent.ThreadPoolExecutor$Worker)
> "sshd-SshServer[67de991c]-nio2-thread-2" - Thread t@124
>java.lang.Thread.State: TIMED_WAITING
>   at java.lang.Object.wait(Native Method)
>   - waiting on <7e9f4eff> (a java.lang.Object)
>   at 
> org.apache.sshd.client.channel.AbstractClientChannel.waitFor(AbstractClientChannel.java:244)
>   at 
> org.apache.sshd.common.forward.DefaultTcpipForwarder$StaticIoHandler.messageReceived(DefaultTcpipForwarder.java:984)
>   at 
> org.apache.sshd.common.io.nio2.Nio2Session.handleReadCycleCompletion(Nio2Session.java:276)
>   at 
> org.apache.sshd.common.io.nio2.Nio2Session$1.onCompleted(Nio2Session.java:256)
>   at 
> org.apache.sshd.common.io.nio2.Nio2Session$1.onCompleted(Nio2Session.java:253)
>   at 
> org.apache.sshd.common.io.nio2.Nio2CompletionHandler.lambda$completed$0(Nio2CompletionHandler.java:38)
>   at 
>

[jira] [Commented] (SSHD-721) deadlock: all nio workers wait to be woken up

2018-04-16 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/SSHD-721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16439383#comment-16439383
 ] 

ASF GitHub Bot commented on SSHD-721:
-

Github user asfgit closed the pull request at:

https://github.com/apache/mina-sshd/pull/53


> deadlock: all nio workers wait to be woken up
> -
>
> Key: SSHD-721
> URL: https://issues.apache.org/jira/browse/SSHD-721
> Project: MINA SSHD
>  Issue Type: Bug
>Affects Versions: 1.3.0, 1.4.0
>Reporter: Markus Rathgeb
>Assignee: Guillaume Nodet
>Priority: Major
> Fix For: 1.8.0
>
>
> I am using sshd-core for a server machine (S) that accepts incoming 
> connections and port forwarding requests.
> There are client machines (C) that run servers that should be accessible by a 
> tunnel to the server.
> On the client machines (C) also an implementation using sshd-core is running 
> that establish the connection to the server (S) and initiate the port 
> forwarding.
> Other clients are using the tunnelled connection to communicate with the 
> servers that are running on the client machines (C).
> Sometimes I realized that no data is transferred anymore (through the 
> tunnels).
> All the worker reside in the waitFor function and no one wakes them up.
> {noformat}
> "sshd-SshServer[67de991c]-nio2-thread-3" - Thread t@125
>java.lang.Thread.State: TIMED_WAITING
>   at java.lang.Object.wait(Native Method)
>   - waiting on <132c6b60> (a java.lang.Object)
>   at 
> org.apache.sshd.client.channel.AbstractClientChannel.waitFor(AbstractClientChannel.java:244)
>   at 
> org.apache.sshd.common.forward.DefaultTcpipForwarder$StaticIoHandler.messageReceived(DefaultTcpipForwarder.java:984)
>   at 
> org.apache.sshd.common.io.nio2.Nio2Session.handleReadCycleCompletion(Nio2Session.java:276)
>   at 
> org.apache.sshd.common.io.nio2.Nio2Session$1.onCompleted(Nio2Session.java:256)
>   at 
> org.apache.sshd.common.io.nio2.Nio2Session$1.onCompleted(Nio2Session.java:253)
>   at 
> org.apache.sshd.common.io.nio2.Nio2CompletionHandler.lambda$completed$0(Nio2CompletionHandler.java:38)
>   at 
> org.apache.sshd.common.io.nio2.Nio2CompletionHandler$$Lambda$45/1071326492.run(Unknown
>  Source)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at 
> org.apache.sshd.common.io.nio2.Nio2CompletionHandler.completed(Nio2CompletionHandler.java:37)
>   at sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:126)
>   at sun.nio.ch.Invoker$2.run(Invoker.java:218)
>   at 
> sun.nio.ch.AsynchronousChannelGroupImpl$1.run(AsynchronousChannelGroupImpl.java:112)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>   at java.lang.Thread.run(Thread.java:745)
>Locked ownable synchronizers:
>   - locked <6d02d7ed> (a java.util.concurrent.ThreadPoolExecutor$Worker)
> "sshd-SshServer[67de991c]-nio2-thread-2" - Thread t@124
>java.lang.Thread.State: TIMED_WAITING
>   at java.lang.Object.wait(Native Method)
>   - waiting on <7e9f4eff> (a java.lang.Object)
>   at 
> org.apache.sshd.client.channel.AbstractClientChannel.waitFor(AbstractClientChannel.java:244)
>   at 
> org.apache.sshd.common.forward.DefaultTcpipForwarder$StaticIoHandler.messageReceived(DefaultTcpipForwarder.java:984)
>   at 
> org.apache.sshd.common.io.nio2.Nio2Session.handleReadCycleCompletion(Nio2Session.java:276)
>   at 
> org.apache.sshd.common.io.nio2.Nio2Session$1.onCompleted(Nio2Session.java:256)
>   at 
> org.apache.sshd.common.io.nio2.Nio2Session$1.onCompleted(Nio2Session.java:253)
>   at 
> org.apache.sshd.common.io.nio2.Nio2CompletionHandler.lambda$completed$0(Nio2CompletionHandler.java:38)
>   at 
> org.apache.sshd.common.io.nio2.Nio2CompletionHandler$$Lambda$45/1071326492.run(Unknown
>  Source)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at 
> org.apache.sshd.common.io.nio2.Nio2CompletionHandler.completed(Nio2CompletionHandler.java:37)
>   at sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:126)
>   at sun.nio.ch.Invoker$2.run(Invoker.java:218)
>   at 
> sun.nio.ch.AsynchronousChannelGroupImpl$1.run(AsynchronousChannelGroupImpl.java:112)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>   at java.lang.Thread.run(Thread.java:745)
>Locked ownable synchronizers:
>   - locked <35fbf3e8> (a java.util.concurrent.ThreadPoolExecutor$Worker)
> "sshd-SshServer[67de991c]-nio2-thread-1" - Thread t@122
>java.lang.Thread.State: TIMED_WAITING
>   at

[jira] [Commented] (SSHD-721) deadlock: all nio workers wait to be woken up

2018-04-16 Thread Guillaume Nodet (JIRA)


[ 
https://issues.apache.org/jira/browse/SSHD-721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16439365#comment-16439365
 ] 

Guillaume Nodet commented on SSHD-721:
--

[~bkuker1] thx, I'm waiting for the test so that I can then commit both...

Thread management could be improved on both the {{TcpipClientChannel}} and 
{{TcpipServerChannel}} by using the async streams ({{ChannelAsyncOutputStream}} 
instead of {{ChannelOutputStream}}).  This would avoid having threads waiting 
for available window space, packet being written, etc..  But that's only an 
improvement, and the current way should work too.  Feel free to raise a JIRA 
about using async stream though.

> deadlock: all nio workers wait to be woken up
> -
>
> Key: SSHD-721
> URL: https://issues.apache.org/jira/browse/SSHD-721
> Project: MINA SSHD
>  Issue Type: Bug
>Affects Versions: 1.3.0, 1.4.0
>Reporter: Markus Rathgeb
>Assignee: Guillaume Nodet
>Priority: Major
>
> I am using sshd-core for a server machine (S) that accepts incoming 
> connections and port forwarding requests.
> There are client machines (C) that run servers that should be accessible by a 
> tunnel to the server.
> On the client machines (C) also an implementation using sshd-core is running 
> that establish the connection to the server (S) and initiate the port 
> forwarding.
> Other clients are using the tunnelled connection to communicate with the 
> servers that are running on the client machines (C).
> Sometimes I realized that no data is transferred anymore (through the 
> tunnels).
> All the worker reside in the waitFor function and no one wakes them up.
> {noformat}
> "sshd-SshServer[67de991c]-nio2-thread-3" - Thread t@125
>java.lang.Thread.State: TIMED_WAITING
>   at java.lang.Object.wait(Native Method)
>   - waiting on <132c6b60> (a java.lang.Object)
>   at 
> org.apache.sshd.client.channel.AbstractClientChannel.waitFor(AbstractClientChannel.java:244)
>   at 
> org.apache.sshd.common.forward.DefaultTcpipForwarder$StaticIoHandler.messageReceived(DefaultTcpipForwarder.java:984)
>   at 
> org.apache.sshd.common.io.nio2.Nio2Session.handleReadCycleCompletion(Nio2Session.java:276)
>   at 
> org.apache.sshd.common.io.nio2.Nio2Session$1.onCompleted(Nio2Session.java:256)
>   at 
> org.apache.sshd.common.io.nio2.Nio2Session$1.onCompleted(Nio2Session.java:253)
>   at 
> org.apache.sshd.common.io.nio2.Nio2CompletionHandler.lambda$completed$0(Nio2CompletionHandler.java:38)
>   at 
> org.apache.sshd.common.io.nio2.Nio2CompletionHandler$$Lambda$45/1071326492.run(Unknown
>  Source)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at 
> org.apache.sshd.common.io.nio2.Nio2CompletionHandler.completed(Nio2CompletionHandler.java:37)
>   at sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:126)
>   at sun.nio.ch.Invoker$2.run(Invoker.java:218)
>   at 
> sun.nio.ch.AsynchronousChannelGroupImpl$1.run(AsynchronousChannelGroupImpl.java:112)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>   at java.lang.Thread.run(Thread.java:745)
>Locked ownable synchronizers:
>   - locked <6d02d7ed> (a java.util.concurrent.ThreadPoolExecutor$Worker)
> "sshd-SshServer[67de991c]-nio2-thread-2" - Thread t@124
>java.lang.Thread.State: TIMED_WAITING
>   at java.lang.Object.wait(Native Method)
>   - waiting on <7e9f4eff> (a java.lang.Object)
>   at 
> org.apache.sshd.client.channel.AbstractClientChannel.waitFor(AbstractClientChannel.java:244)
>   at 
> org.apache.sshd.common.forward.DefaultTcpipForwarder$StaticIoHandler.messageReceived(DefaultTcpipForwarder.java:984)
>   at 
> org.apache.sshd.common.io.nio2.Nio2Session.handleReadCycleCompletion(Nio2Session.java:276)
>   at 
> org.apache.sshd.common.io.nio2.Nio2Session$1.onCompleted(Nio2Session.java:256)
>   at 
> org.apache.sshd.common.io.nio2.Nio2Session$1.onCompleted(Nio2Session.java:253)
>   at 
> org.apache.sshd.common.io.nio2.Nio2CompletionHandler.lambda$completed$0(Nio2CompletionHandler.java:38)
>   at 
> org.apache.sshd.common.io.nio2.Nio2CompletionHandler$$Lambda$45/1071326492.run(Unknown
>  Source)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at 
> org.apache.sshd.common.io.nio2.Nio2CompletionHandler.completed(Nio2CompletionHandler.java:37)
>   at sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:126)
>   at sun.nio.ch.Invoker$2.run(Invoker.java:218)
>   at 
> sun.nio.ch.AsynchronousChannelGroupImpl$1.run(AsynchronousChannelGroupImpl.java:112)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>   at 
>

[jira] [Commented] (SSHD-721) deadlock: all nio workers wait to be woken up

2018-04-16 Thread Bill Kuker (JIRA)


[ 
https://issues.apache.org/jira/browse/SSHD-721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16439336#comment-16439336
 ] 

Bill Kuker commented on SSHD-721:
-

[~gnt], thanks so much, that patch seems to have a positive effect (/) :). It 
probably would have taken me all week to get there on my own, I am not really 
familiar with this code base.

I'll button up the test case I have for this (after my coffee and morning 
meeting), really make sure it's happy, and pass it along.

I also wanted to ask about 
org.apache.sshd.server.forward.TcpipServerChannel.doInit(Buffer) line ~153:  
{{// TODO: revisit for better threading. Use async io ?}} I had been wondering 
if that was related.

> deadlock: all nio workers wait to be woken up
> -
>
> Key: SSHD-721
> URL: https://issues.apache.org/jira/browse/SSHD-721
> Project: MINA SSHD
>  Issue Type: Bug
>Affects Versions: 1.3.0, 1.4.0
>Reporter: Markus Rathgeb
>Priority: Major
>
> I am using sshd-core for a server machine (S) that accepts incoming 
> connections and port forwarding requests.
> There are client machines (C) that run servers that should be accessible by a 
> tunnel to the server.
> On the client machines (C) also an implementation using sshd-core is running 
> that establish the connection to the server (S) and initiate the port 
> forwarding.
> Other clients are using the tunnelled connection to communicate with the 
> servers that are running on the client machines (C).
> Sometimes I realized that no data is transferred anymore (through the 
> tunnels).
> All the worker reside in the waitFor function and no one wakes them up.
> {noformat}
> "sshd-SshServer[67de991c]-nio2-thread-3" - Thread t@125
>java.lang.Thread.State: TIMED_WAITING
>   at java.lang.Object.wait(Native Method)
>   - waiting on <132c6b60> (a java.lang.Object)
>   at 
> org.apache.sshd.client.channel.AbstractClientChannel.waitFor(AbstractClientChannel.java:244)
>   at 
> org.apache.sshd.common.forward.DefaultTcpipForwarder$StaticIoHandler.messageReceived(DefaultTcpipForwarder.java:984)
>   at 
> org.apache.sshd.common.io.nio2.Nio2Session.handleReadCycleCompletion(Nio2Session.java:276)
>   at 
> org.apache.sshd.common.io.nio2.Nio2Session$1.onCompleted(Nio2Session.java:256)
>   at 
> org.apache.sshd.common.io.nio2.Nio2Session$1.onCompleted(Nio2Session.java:253)
>   at 
> org.apache.sshd.common.io.nio2.Nio2CompletionHandler.lambda$completed$0(Nio2CompletionHandler.java:38)
>   at 
> org.apache.sshd.common.io.nio2.Nio2CompletionHandler$$Lambda$45/1071326492.run(Unknown
>  Source)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at 
> org.apache.sshd.common.io.nio2.Nio2CompletionHandler.completed(Nio2CompletionHandler.java:37)
>   at sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:126)
>   at sun.nio.ch.Invoker$2.run(Invoker.java:218)
>   at 
> sun.nio.ch.AsynchronousChannelGroupImpl$1.run(AsynchronousChannelGroupImpl.java:112)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>   at java.lang.Thread.run(Thread.java:745)
>Locked ownable synchronizers:
>   - locked <6d02d7ed> (a java.util.concurrent.ThreadPoolExecutor$Worker)
> "sshd-SshServer[67de991c]-nio2-thread-2" - Thread t@124
>java.lang.Thread.State: TIMED_WAITING
>   at java.lang.Object.wait(Native Method)
>   - waiting on <7e9f4eff> (a java.lang.Object)
>   at 
> org.apache.sshd.client.channel.AbstractClientChannel.waitFor(AbstractClientChannel.java:244)
>   at 
> org.apache.sshd.common.forward.DefaultTcpipForwarder$StaticIoHandler.messageReceived(DefaultTcpipForwarder.java:984)
>   at 
> org.apache.sshd.common.io.nio2.Nio2Session.handleReadCycleCompletion(Nio2Session.java:276)
>   at 
> org.apache.sshd.common.io.nio2.Nio2Session$1.onCompleted(Nio2Session.java:256)
>   at 
> org.apache.sshd.common.io.nio2.Nio2Session$1.onCompleted(Nio2Session.java:253)
>   at 
> org.apache.sshd.common.io.nio2.Nio2CompletionHandler.lambda$completed$0(Nio2CompletionHandler.java:38)
>   at 
> org.apache.sshd.common.io.nio2.Nio2CompletionHandler$$Lambda$45/1071326492.run(Unknown
>  Source)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at 
> org.apache.sshd.common.io.nio2.Nio2CompletionHandler.completed(Nio2CompletionHandler.java:37)
>   at sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:126)
>   at sun.nio.ch.Invoker$2.run(Invoker.java:218)
>   at 
> sun.nio.ch.AsynchronousChannelGroupImpl$1.run(AsynchronousChannelGroupImpl.java:112)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>   at 
>

[jira] [Commented] (SSHD-721) deadlock: all nio workers wait to be woken up

2018-04-16 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/SSHD-721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16439164#comment-16439164
 ] 

ASF GitHub Bot commented on SSHD-721:
-

GitHub user gnodet opened a pull request:

https://github.com/apache/mina-sshd/pull/53

[SSHD-721] I/O workers exhaustion in tcpip forwarding



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gnodet/mina-sshd SSHD-721

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/mina-sshd/pull/53.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #53


commit ca578eb1c74d94ad89e2d4ab0d94489eace7a0c6
Author: Guillaume Nodet 
Date:   2018-04-15T19:01:09Z

[SSHD-721] I/O workers exhaustion in tcpip forwarding




> deadlock: all nio workers wait to be woken up
> -
>
> Key: SSHD-721
> URL: https://issues.apache.org/jira/browse/SSHD-721
> Project: MINA SSHD
>  Issue Type: Bug
>Affects Versions: 1.3.0, 1.4.0
>Reporter: Markus Rathgeb
>Priority: Major
>
> I am using sshd-core for a server machine (S) that accepts incoming 
> connections and port forwarding requests.
> There are client machines (C) that run servers that should be accessible by a 
> tunnel to the server.
> On the client machines (C) also an implementation using sshd-core is running 
> that establish the connection to the server (S) and initiate the port 
> forwarding.
> Other clients are using the tunnelled connection to communicate with the 
> servers that are running on the client machines (C).
> Sometimes I realized that no data is transferred anymore (through the 
> tunnels).
> All the worker reside in the waitFor function and no one wakes them up.
> {noformat}
> "sshd-SshServer[67de991c]-nio2-thread-3" - Thread t@125
>java.lang.Thread.State: TIMED_WAITING
>   at java.lang.Object.wait(Native Method)
>   - waiting on <132c6b60> (a java.lang.Object)
>   at 
> org.apache.sshd.client.channel.AbstractClientChannel.waitFor(AbstractClientChannel.java:244)
>   at 
> org.apache.sshd.common.forward.DefaultTcpipForwarder$StaticIoHandler.messageReceived(DefaultTcpipForwarder.java:984)
>   at 
> org.apache.sshd.common.io.nio2.Nio2Session.handleReadCycleCompletion(Nio2Session.java:276)
>   at 
> org.apache.sshd.common.io.nio2.Nio2Session$1.onCompleted(Nio2Session.java:256)
>   at 
> org.apache.sshd.common.io.nio2.Nio2Session$1.onCompleted(Nio2Session.java:253)
>   at 
> org.apache.sshd.common.io.nio2.Nio2CompletionHandler.lambda$completed$0(Nio2CompletionHandler.java:38)
>   at 
> org.apache.sshd.common.io.nio2.Nio2CompletionHandler$$Lambda$45/1071326492.run(Unknown
>  Source)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at 
> org.apache.sshd.common.io.nio2.Nio2CompletionHandler.completed(Nio2CompletionHandler.java:37)
>   at sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:126)
>   at sun.nio.ch.Invoker$2.run(Invoker.java:218)
>   at 
> sun.nio.ch.AsynchronousChannelGroupImpl$1.run(AsynchronousChannelGroupImpl.java:112)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>   at java.lang.Thread.run(Thread.java:745)
>Locked ownable synchronizers:
>   - locked <6d02d7ed> (a java.util.concurrent.ThreadPoolExecutor$Worker)
> "sshd-SshServer[67de991c]-nio2-thread-2" - Thread t@124
>java.lang.Thread.State: TIMED_WAITING
>   at java.lang.Object.wait(Native Method)
>   - waiting on <7e9f4eff> (a java.lang.Object)
>   at 
> org.apache.sshd.client.channel.AbstractClientChannel.waitFor(AbstractClientChannel.java:244)
>   at 
> org.apache.sshd.common.forward.DefaultTcpipForwarder$StaticIoHandler.messageReceived(DefaultTcpipForwarder.java:984)
>   at 
> org.apache.sshd.common.io.nio2.Nio2Session.handleReadCycleCompletion(Nio2Session.java:276)
>   at 
> org.apache.sshd.common.io.nio2.Nio2Session$1.onCompleted(Nio2Session.java:256)
>   at 
> org.apache.sshd.common.io.nio2.Nio2Session$1.onCompleted(Nio2Session.java:253)
>   at 
> org.apache.sshd.common.io.nio2.Nio2CompletionHandler.lambda$completed$0(Nio2CompletionHandler.java:38)
>   at 
> org.apache.sshd.common.io.nio2.Nio2CompletionHandler$$Lambda$45/1071326492.run(Unknown
>  Source)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at 
> org.apache.sshd.common.io.nio2.Nio2CompletionHandler.completed(Nio2CompletionHandler.java:37)
>   at sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:126)
>   at sun.nio.ch.Invoker$2.run(Invoker.java:218)
>

[jira] [Commented] (SSHD-721) deadlock: all nio workers wait to be woken up

2018-04-15 Thread Guillaume Nodet (JIRA)


[ 
https://issues.apache.org/jira/browse/SSHD-721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16438803#comment-16438803
 ] 

Guillaume Nodet commented on SSHD-721:
--

I think the code needs to be fixed: you can't perform an infinite wait on the 
channel from a worker thread or you'll run into this very issue: i.e. thread 
pool exhaustion.

Could you try the following patch :
{code}
diff --git 
a/sshd-core/src/main/java/org/apache/sshd/common/forward/DefaultForwardingFilter.java
 
b/sshd-core/src/main/java/org/apache/sshd/common/forward/DefaultForwardingFilter.java
index 79364152..751f73a3 100644
--- 
a/sshd-core/src/main/java/org/apache/sshd/common/forward/DefaultForwardingFilter.java
+++ 
b/sshd-core/src/main/java/org/apache/sshd/common/forward/DefaultForwardingFilter.java
@@ -37,6 +37,7 @@ import java.util.concurrent.TimeUnit;
 import java.util.concurrent.atomic.AtomicLong;
 
 import org.apache.sshd.client.channel.ClientChannelEvent;
+import org.apache.sshd.client.future.OpenFuture;
 import org.apache.sshd.common.Closeable;
 import org.apache.sshd.common.Factory;
 import org.apache.sshd.common.FactoryManager;
@@ -979,15 +980,22 @@ public class DefaultForwardingFilter
   session, channel, totalMessages, 
message.available());
 }
 
-Collection result = 
channel.waitFor(STATIC_IO_MSG_RECEIVED_EVENTS, Long.MAX_VALUE);
-if (traceEnabled) {
-log.trace("messageReceived({}) channel={}, count={}, len={} 
wait result: {}",
-  session, channel, totalMessages, 
message.available(), result);
+OpenFuture future = channel.getOpenFuture();
+if (future.isOpened()) {
+OutputStream outputStream = channel.getInvertedIn();
+outputStream.write(buffer.array(), buffer.rpos(), 
buffer.available());
+outputStream.flush();
+} else {
+future.addListener(f -> {
+try {
+OutputStream outputStream = channel.getInvertedIn();
+outputStream.write(buffer.array(), buffer.rpos(), 
buffer.available());
+outputStream.flush();
+} catch (IOException e) {
+channel.getSession().exceptionCaught(e);
+}
+});
 }
-
-OutputStream outputStream = channel.getInvertedIn();
-outputStream.write(buffer.array(), buffer.rpos(), 
buffer.available());
-outputStream.flush();
 }
 
 @Override
{code}

> deadlock: all nio workers wait to be woken up
> -
>
> Key: SSHD-721
> URL: https://issues.apache.org/jira/browse/SSHD-721
> Project: MINA SSHD
>  Issue Type: Bug
>Affects Versions: 1.3.0, 1.4.0
>Reporter: Markus Rathgeb
>Priority: Major
>
> I am using sshd-core for a server machine (S) that accepts incoming 
> connections and port forwarding requests.
> There are client machines (C) that run servers that should be accessible by a 
> tunnel to the server.
> On the client machines (C) also an implementation using sshd-core is running 
> that establish the connection to the server (S) and initiate the port 
> forwarding.
> Other clients are using the tunnelled connection to communicate with the 
> servers that are running on the client machines (C).
> Sometimes I realized that no data is transferred anymore (through the 
> tunnels).
> All the worker reside in the waitFor function and no one wakes them up.
> {noformat}
> "sshd-SshServer[67de991c]-nio2-thread-3" - Thread t@125
>java.lang.Thread.State: TIMED_WAITING
>   at java.lang.Object.wait(Native Method)
>   - waiting on <132c6b60> (a java.lang.Object)
>   at 
> org.apache.sshd.client.channel.AbstractClientChannel.waitFor(AbstractClientChannel.java:244)
>   at 
> org.apache.sshd.common.forward.DefaultTcpipForwarder$StaticIoHandler.messageReceived(DefaultTcpipForwarder.java:984)
>   at 
> org.apache.sshd.common.io.nio2.Nio2Session.handleReadCycleCompletion(Nio2Session.java:276)
>   at 
> org.apache.sshd.common.io.nio2.Nio2Session$1.onCompleted(Nio2Session.java:256)
>   at 
> org.apache.sshd.common.io.nio2.Nio2Session$1.onCompleted(Nio2Session.java:253)
>   at 
> org.apache.sshd.common.io.nio2.Nio2CompletionHandler.lambda$completed$0(Nio2CompletionHandler.java:38)
>   at 
> org.apache.sshd.common.io.nio2.Nio2CompletionHandler$$Lambda$45/1071326492.run(Unknown
>  Source)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at 
> org.apache.sshd.common.io.nio2.Nio2CompletionHandler.completed(Nio2CompletionHandler.java:37)
>   at sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:126)
>   at sun.nio.ch.Invoker$2.run(Invoker.java:218)
>

[jira] [Commented] (SSHD-721) deadlock: all nio workers wait to be woken up

2018-04-14 Thread Bill Kuker (JIRA)


[ 
https://issues.apache.org/jira/browse/SSHD-721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16438468#comment-16438468
 ] 

Bill Kuker commented on SSHD-721:
-

[~johnnyv] That is correct. I do not think switching over to an unbounded 
thread pool is the right solution, but it is a temporary workaround for the bug 
for me.

If whichever operation is blocking could be converted to non-blocking that 
could fix the issue, or running these 2 different kinds of operation in 2 
different thread pools could also fix the issue.

> deadlock: all nio workers wait to be woken up
> -
>
> Key: SSHD-721
> URL: https://issues.apache.org/jira/browse/SSHD-721
> Project: MINA SSHD
>  Issue Type: Bug
>Affects Versions: 1.3.0, 1.4.0
>Reporter: Markus Rathgeb
>Priority: Major
>
> I am using sshd-core for a server machine (S) that accepts incoming 
> connections and port forwarding requests.
> There are client machines (C) that run servers that should be accessible by a 
> tunnel to the server.
> On the client machines (C) also an implementation using sshd-core is running 
> that establish the connection to the server (S) and initiate the port 
> forwarding.
> Other clients are using the tunnelled connection to communicate with the 
> servers that are running on the client machines (C).
> Sometimes I realized that no data is transferred anymore (through the 
> tunnels).
> All the worker reside in the waitFor function and no one wakes them up.
> {noformat}
> "sshd-SshServer[67de991c]-nio2-thread-3" - Thread t@125
>java.lang.Thread.State: TIMED_WAITING
>   at java.lang.Object.wait(Native Method)
>   - waiting on <132c6b60> (a java.lang.Object)
>   at 
> org.apache.sshd.client.channel.AbstractClientChannel.waitFor(AbstractClientChannel.java:244)
>   at 
> org.apache.sshd.common.forward.DefaultTcpipForwarder$StaticIoHandler.messageReceived(DefaultTcpipForwarder.java:984)
>   at 
> org.apache.sshd.common.io.nio2.Nio2Session.handleReadCycleCompletion(Nio2Session.java:276)
>   at 
> org.apache.sshd.common.io.nio2.Nio2Session$1.onCompleted(Nio2Session.java:256)
>   at 
> org.apache.sshd.common.io.nio2.Nio2Session$1.onCompleted(Nio2Session.java:253)
>   at 
> org.apache.sshd.common.io.nio2.Nio2CompletionHandler.lambda$completed$0(Nio2CompletionHandler.java:38)
>   at 
> org.apache.sshd.common.io.nio2.Nio2CompletionHandler$$Lambda$45/1071326492.run(Unknown
>  Source)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at 
> org.apache.sshd.common.io.nio2.Nio2CompletionHandler.completed(Nio2CompletionHandler.java:37)
>   at sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:126)
>   at sun.nio.ch.Invoker$2.run(Invoker.java:218)
>   at 
> sun.nio.ch.AsynchronousChannelGroupImpl$1.run(AsynchronousChannelGroupImpl.java:112)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>   at java.lang.Thread.run(Thread.java:745)
>Locked ownable synchronizers:
>   - locked <6d02d7ed> (a java.util.concurrent.ThreadPoolExecutor$Worker)
> "sshd-SshServer[67de991c]-nio2-thread-2" - Thread t@124
>java.lang.Thread.State: TIMED_WAITING
>   at java.lang.Object.wait(Native Method)
>   - waiting on <7e9f4eff> (a java.lang.Object)
>   at 
> org.apache.sshd.client.channel.AbstractClientChannel.waitFor(AbstractClientChannel.java:244)
>   at 
> org.apache.sshd.common.forward.DefaultTcpipForwarder$StaticIoHandler.messageReceived(DefaultTcpipForwarder.java:984)
>   at 
> org.apache.sshd.common.io.nio2.Nio2Session.handleReadCycleCompletion(Nio2Session.java:276)
>   at 
> org.apache.sshd.common.io.nio2.Nio2Session$1.onCompleted(Nio2Session.java:256)
>   at 
> org.apache.sshd.common.io.nio2.Nio2Session$1.onCompleted(Nio2Session.java:253)
>   at 
> org.apache.sshd.common.io.nio2.Nio2CompletionHandler.lambda$completed$0(Nio2CompletionHandler.java:38)
>   at 
> org.apache.sshd.common.io.nio2.Nio2CompletionHandler$$Lambda$45/1071326492.run(Unknown
>  Source)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at 
> org.apache.sshd.common.io.nio2.Nio2CompletionHandler.completed(Nio2CompletionHandler.java:37)
>   at sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:126)
>   at sun.nio.ch.Invoker$2.run(Invoker.java:218)
>   at 
> sun.nio.ch.AsynchronousChannelGroupImpl$1.run(AsynchronousChannelGroupImpl.java:112)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>   at java.lang.Thread.run(Thread.java:745)
>Locked ownable synchronizers:
>   - locked

[jira] [Commented] (SSHD-721) deadlock: all nio workers wait to be woken up

2018-04-14 Thread Jonathan Valliere (JIRA)


[ 
https://issues.apache.org/jira/browse/SSHD-721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16438378#comment-16438378
 ] 

Jonathan Valliere commented on SSHD-721:


The {{newCachedThreadPool}} uses a {{SynchronousQueue}} which basically holds 
only one object in the queue and blocks inserts until the queue is empty.  This 
is fixing your problem because {{newCachedThreadPool}} has 
{{Integer.MAX_VALUE}} number of threads.

> deadlock: all nio workers wait to be woken up
> -
>
> Key: SSHD-721
> URL: https://issues.apache.org/jira/browse/SSHD-721
> Project: MINA SSHD
>  Issue Type: Bug
>Affects Versions: 1.3.0, 1.4.0
>Reporter: Markus Rathgeb
>Priority: Major
>
> I am using sshd-core for a server machine (S) that accepts incoming 
> connections and port forwarding requests.
> There are client machines (C) that run servers that should be accessible by a 
> tunnel to the server.
> On the client machines (C) also an implementation using sshd-core is running 
> that establish the connection to the server (S) and initiate the port 
> forwarding.
> Other clients are using the tunnelled connection to communicate with the 
> servers that are running on the client machines (C).
> Sometimes I realized that no data is transferred anymore (through the 
> tunnels).
> All the worker reside in the waitFor function and no one wakes them up.
> {noformat}
> "sshd-SshServer[67de991c]-nio2-thread-3" - Thread t@125
>java.lang.Thread.State: TIMED_WAITING
>   at java.lang.Object.wait(Native Method)
>   - waiting on <132c6b60> (a java.lang.Object)
>   at 
> org.apache.sshd.client.channel.AbstractClientChannel.waitFor(AbstractClientChannel.java:244)
>   at 
> org.apache.sshd.common.forward.DefaultTcpipForwarder$StaticIoHandler.messageReceived(DefaultTcpipForwarder.java:984)
>   at 
> org.apache.sshd.common.io.nio2.Nio2Session.handleReadCycleCompletion(Nio2Session.java:276)
>   at 
> org.apache.sshd.common.io.nio2.Nio2Session$1.onCompleted(Nio2Session.java:256)
>   at 
> org.apache.sshd.common.io.nio2.Nio2Session$1.onCompleted(Nio2Session.java:253)
>   at 
> org.apache.sshd.common.io.nio2.Nio2CompletionHandler.lambda$completed$0(Nio2CompletionHandler.java:38)
>   at 
> org.apache.sshd.common.io.nio2.Nio2CompletionHandler$$Lambda$45/1071326492.run(Unknown
>  Source)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at 
> org.apache.sshd.common.io.nio2.Nio2CompletionHandler.completed(Nio2CompletionHandler.java:37)
>   at sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:126)
>   at sun.nio.ch.Invoker$2.run(Invoker.java:218)
>   at 
> sun.nio.ch.AsynchronousChannelGroupImpl$1.run(AsynchronousChannelGroupImpl.java:112)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>   at java.lang.Thread.run(Thread.java:745)
>Locked ownable synchronizers:
>   - locked <6d02d7ed> (a java.util.concurrent.ThreadPoolExecutor$Worker)
> "sshd-SshServer[67de991c]-nio2-thread-2" - Thread t@124
>java.lang.Thread.State: TIMED_WAITING
>   at java.lang.Object.wait(Native Method)
>   - waiting on <7e9f4eff> (a java.lang.Object)
>   at 
> org.apache.sshd.client.channel.AbstractClientChannel.waitFor(AbstractClientChannel.java:244)
>   at 
> org.apache.sshd.common.forward.DefaultTcpipForwarder$StaticIoHandler.messageReceived(DefaultTcpipForwarder.java:984)
>   at 
> org.apache.sshd.common.io.nio2.Nio2Session.handleReadCycleCompletion(Nio2Session.java:276)
>   at 
> org.apache.sshd.common.io.nio2.Nio2Session$1.onCompleted(Nio2Session.java:256)
>   at 
> org.apache.sshd.common.io.nio2.Nio2Session$1.onCompleted(Nio2Session.java:253)
>   at 
> org.apache.sshd.common.io.nio2.Nio2CompletionHandler.lambda$completed$0(Nio2CompletionHandler.java:38)
>   at 
> org.apache.sshd.common.io.nio2.Nio2CompletionHandler$$Lambda$45/1071326492.run(Unknown
>  Source)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at 
> org.apache.sshd.common.io.nio2.Nio2CompletionHandler.completed(Nio2CompletionHandler.java:37)
>   at sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:126)
>   at sun.nio.ch.Invoker$2.run(Invoker.java:218)
>   at 
> sun.nio.ch.AsynchronousChannelGroupImpl$1.run(AsynchronousChannelGroupImpl.java:112)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>   at java.lang.Thread.run(Thread.java:745)
>Locked ownable synchronizers:
>   - locked <35fbf3e8> (a java.util.concurrent.ThreadPoolExecutor$Worker)
>

[jira] [Commented] (SSHD-721) deadlock: all nio workers wait to be woken up

2018-04-14 Thread Bill Kuker (JIRA)


[ 
https://issues.apache.org/jira/browse/SSHD-721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16438353#comment-16438353
 ] 

Bill Kuker commented on SSHD-721:
-

I am zeroing in on reproducing it...

(In the following steps "web browser" and "web server" are just tiny java 
functions playing those roles)
 # Create an SSH Server, SSH Client and a port forward listening on the SSH 
Server.
 # Configure the SSH Server with *N* nio worker threads.
 # Start the "Web Server" that accepts a connection, reads some data from the 
client, writes some data to the client, closes the connection.
 # Launch *W* "Web Browsers" that each open a connection to the Web Server via 
the port forward listening on the SSH Server.
 # Use a cyclic barrier or similar to make all *W* Web Browsers connect and 
send their request simultaneously. 

(/) When *W < N* everything works fine.

(x) When *W >= N* then the SSHD Server completely stops working. It will never 
recover (Unless some outside force kills some of the processes or tcp 
connections).

My guess is that the SSH Server can handle N-1 port forwards connecting at 
once, and the 1 extra worker thread is used for communication to the SSH 
Client. Once all worker threads are handling new connections to a port forward, 
no thread is available to make progress on communication to the sshd client.

I will be providing a Junit test case once I have cleaned up my code.

Replacing the nio2 thread pool, which is created by 
ThreadUtils::newFixedThreadPool() with a pool created by 
ThreadUtils::newCachedThreadPool() seems to fix the problem, although I am not 
suggesting that as the final fix.

If my guess is correct, my suggestion for the fix would be different worker 
pools for SSH Server / SSH Client communication and for handling the port 
forwards. This would allow SSHD to operate when the number of simultaneous port 
forwarded connections being established is larger than the number of threads in 
either pool, and also allow users to tune these two thread pools separately 
depending on their usage patterns. 

 

> deadlock: all nio workers wait to be woken up
> -
>
> Key: SSHD-721
> URL: https://issues.apache.org/jira/browse/SSHD-721
> Project: MINA SSHD
>  Issue Type: Bug
>Affects Versions: 1.3.0, 1.4.0
>Reporter: Markus Rathgeb
>Priority: Major
>
> I am using sshd-core for a server machine (S) that accepts incoming 
> connections and port forwarding requests.
> There are client machines (C) that run servers that should be accessible by a 
> tunnel to the server.
> On the client machines (C) also an implementation using sshd-core is running 
> that establish the connection to the server (S) and initiate the port 
> forwarding.
> Other clients are using the tunnelled connection to communicate with the 
> servers that are running on the client machines (C).
> Sometimes I realized that no data is transferred anymore (through the 
> tunnels).
> All the worker reside in the waitFor function and no one wakes them up.
> {noformat}
> "sshd-SshServer[67de991c]-nio2-thread-3" - Thread t@125
>java.lang.Thread.State: TIMED_WAITING
>   at java.lang.Object.wait(Native Method)
>   - waiting on <132c6b60> (a java.lang.Object)
>   at 
> org.apache.sshd.client.channel.AbstractClientChannel.waitFor(AbstractClientChannel.java:244)
>   at 
> org.apache.sshd.common.forward.DefaultTcpipForwarder$StaticIoHandler.messageReceived(DefaultTcpipForwarder.java:984)
>   at 
> org.apache.sshd.common.io.nio2.Nio2Session.handleReadCycleCompletion(Nio2Session.java:276)
>   at 
> org.apache.sshd.common.io.nio2.Nio2Session$1.onCompleted(Nio2Session.java:256)
>   at 
> org.apache.sshd.common.io.nio2.Nio2Session$1.onCompleted(Nio2Session.java:253)
>   at 
> org.apache.sshd.common.io.nio2.Nio2CompletionHandler.lambda$completed$0(Nio2CompletionHandler.java:38)
>   at 
> org.apache.sshd.common.io.nio2.Nio2CompletionHandler$$Lambda$45/1071326492.run(Unknown
>  Source)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at 
> org.apache.sshd.common.io.nio2.Nio2CompletionHandler.completed(Nio2CompletionHandler.java:37)
>   at sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:126)
>   at sun.nio.ch.Invoker$2.run(Invoker.java:218)
>   at 
> sun.nio.ch.AsynchronousChannelGroupImpl$1.run(AsynchronousChannelGroupImpl.java:112)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>   at java.lang.Thread.run(Thread.java:745)
>Locked ownable synchronizers:
>   - locked <6d02d7ed> (a java.util.concurrent.ThreadPoolExecutor$Worker)
> "sshd-SshServer[67de991c]-nio2-thread-2" - Thread t@124
>java.lang.Thread.State: TIMED_WAITING

[jira] [Commented] (SSHD-721) deadlock: all nio workers wait to be woken up

2018-04-13 Thread Jonathan Valliere (JIRA)


[ 
https://issues.apache.org/jira/browse/SSHD-721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16437302#comment-16437302
 ] 

Jonathan Valliere commented on SSHD-721:


{quote}
Once all clients with port forwards have killed themselves sshd begins 
operating normally.
{quote}

This makes me think that everything works correctly and your problem lies in 
the fact that port forwarding must be a blocking operation.

[~gnt] any input?

> deadlock: all nio workers wait to be woken up
> -
>
> Key: SSHD-721
> URL: https://issues.apache.org/jira/browse/SSHD-721
> Project: MINA SSHD
>  Issue Type: Bug
>Affects Versions: 1.3.0, 1.4.0
>Reporter: Markus Rathgeb
>Priority: Major
>
> I am using sshd-core for a server machine (S) that accepts incoming 
> connections and port forwarding requests.
> There are client machines (C) that run servers that should be accessible by a 
> tunnel to the server.
> On the client machines (C) also an implementation using sshd-core is running 
> that establish the connection to the server (S) and initiate the port 
> forwarding.
> Other clients are using the tunnelled connection to communicate with the 
> servers that are running on the client machines (C).
> Sometimes I realized that no data is transferred anymore (through the 
> tunnels).
> All the worker reside in the waitFor function and no one wakes them up.
> {noformat}
> "sshd-SshServer[67de991c]-nio2-thread-3" - Thread t@125
>java.lang.Thread.State: TIMED_WAITING
>   at java.lang.Object.wait(Native Method)
>   - waiting on <132c6b60> (a java.lang.Object)
>   at 
> org.apache.sshd.client.channel.AbstractClientChannel.waitFor(AbstractClientChannel.java:244)
>   at 
> org.apache.sshd.common.forward.DefaultTcpipForwarder$StaticIoHandler.messageReceived(DefaultTcpipForwarder.java:984)
>   at 
> org.apache.sshd.common.io.nio2.Nio2Session.handleReadCycleCompletion(Nio2Session.java:276)
>   at 
> org.apache.sshd.common.io.nio2.Nio2Session$1.onCompleted(Nio2Session.java:256)
>   at 
> org.apache.sshd.common.io.nio2.Nio2Session$1.onCompleted(Nio2Session.java:253)
>   at 
> org.apache.sshd.common.io.nio2.Nio2CompletionHandler.lambda$completed$0(Nio2CompletionHandler.java:38)
>   at 
> org.apache.sshd.common.io.nio2.Nio2CompletionHandler$$Lambda$45/1071326492.run(Unknown
>  Source)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at 
> org.apache.sshd.common.io.nio2.Nio2CompletionHandler.completed(Nio2CompletionHandler.java:37)
>   at sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:126)
>   at sun.nio.ch.Invoker$2.run(Invoker.java:218)
>   at 
> sun.nio.ch.AsynchronousChannelGroupImpl$1.run(AsynchronousChannelGroupImpl.java:112)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>   at java.lang.Thread.run(Thread.java:745)
>Locked ownable synchronizers:
>   - locked <6d02d7ed> (a java.util.concurrent.ThreadPoolExecutor$Worker)
> "sshd-SshServer[67de991c]-nio2-thread-2" - Thread t@124
>java.lang.Thread.State: TIMED_WAITING
>   at java.lang.Object.wait(Native Method)
>   - waiting on <7e9f4eff> (a java.lang.Object)
>   at 
> org.apache.sshd.client.channel.AbstractClientChannel.waitFor(AbstractClientChannel.java:244)
>   at 
> org.apache.sshd.common.forward.DefaultTcpipForwarder$StaticIoHandler.messageReceived(DefaultTcpipForwarder.java:984)
>   at 
> org.apache.sshd.common.io.nio2.Nio2Session.handleReadCycleCompletion(Nio2Session.java:276)
>   at 
> org.apache.sshd.common.io.nio2.Nio2Session$1.onCompleted(Nio2Session.java:256)
>   at 
> org.apache.sshd.common.io.nio2.Nio2Session$1.onCompleted(Nio2Session.java:253)
>   at 
> org.apache.sshd.common.io.nio2.Nio2CompletionHandler.lambda$completed$0(Nio2CompletionHandler.java:38)
>   at 
> org.apache.sshd.common.io.nio2.Nio2CompletionHandler$$Lambda$45/1071326492.run(Unknown
>  Source)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at 
> org.apache.sshd.common.io.nio2.Nio2CompletionHandler.completed(Nio2CompletionHandler.java:37)
>   at sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:126)
>   at sun.nio.ch.Invoker$2.run(Invoker.java:218)
>   at 
> sun.nio.ch.AsynchronousChannelGroupImpl$1.run(AsynchronousChannelGroupImpl.java:112)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>   at java.lang.Thread.run(Thread.java:745)
>Locked ownable synchronizers:
>   - locked <35fbf3e8> (a java.util.concurrent.ThreadPoolExecutor$Worker)
>

[jira] [Commented] (SSHD-721) deadlock: all nio workers wait to be woken up

2018-04-13 Thread Bill Kuker (JIRA)


[ 
https://issues.apache.org/jira/browse/SSHD-721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16437253#comment-16437253
 ] 

Bill Kuker commented on SSHD-721:
-

You know, as I am waking up this morning, I do remember noticing that in the 
change from 0.3.0 to 1.2.0 the port forwards and the client-server 
communication moved onto the same thread pool.

{{web browser <-> ssh server <-> ssh client <-> web server}}

Could it be that simple: If all nio workers are working on sending or receiving 
data to / from the web browser, no worker is available to send that data to the 
ssh client?

> deadlock: all nio workers wait to be woken up
> -
>
> Key: SSHD-721
> URL: https://issues.apache.org/jira/browse/SSHD-721
> Project: MINA SSHD
>  Issue Type: Bug
>Affects Versions: 1.3.0, 1.4.0
>Reporter: Markus Rathgeb
>Priority: Major
>
> I am using sshd-core for a server machine (S) that accepts incoming 
> connections and port forwarding requests.
> There are client machines (C) that run servers that should be accessible by a 
> tunnel to the server.
> On the client machines (C) also an implementation using sshd-core is running 
> that establish the connection to the server (S) and initiate the port 
> forwarding.
> Other clients are using the tunnelled connection to communicate with the 
> servers that are running on the client machines (C).
> Sometimes I realized that no data is transferred anymore (through the 
> tunnels).
> All the worker reside in the waitFor function and no one wakes them up.
> {noformat}
> "sshd-SshServer[67de991c]-nio2-thread-3" - Thread t@125
>java.lang.Thread.State: TIMED_WAITING
>   at java.lang.Object.wait(Native Method)
>   - waiting on <132c6b60> (a java.lang.Object)
>   at 
> org.apache.sshd.client.channel.AbstractClientChannel.waitFor(AbstractClientChannel.java:244)
>   at 
> org.apache.sshd.common.forward.DefaultTcpipForwarder$StaticIoHandler.messageReceived(DefaultTcpipForwarder.java:984)
>   at 
> org.apache.sshd.common.io.nio2.Nio2Session.handleReadCycleCompletion(Nio2Session.java:276)
>   at 
> org.apache.sshd.common.io.nio2.Nio2Session$1.onCompleted(Nio2Session.java:256)
>   at 
> org.apache.sshd.common.io.nio2.Nio2Session$1.onCompleted(Nio2Session.java:253)
>   at 
> org.apache.sshd.common.io.nio2.Nio2CompletionHandler.lambda$completed$0(Nio2CompletionHandler.java:38)
>   at 
> org.apache.sshd.common.io.nio2.Nio2CompletionHandler$$Lambda$45/1071326492.run(Unknown
>  Source)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at 
> org.apache.sshd.common.io.nio2.Nio2CompletionHandler.completed(Nio2CompletionHandler.java:37)
>   at sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:126)
>   at sun.nio.ch.Invoker$2.run(Invoker.java:218)
>   at 
> sun.nio.ch.AsynchronousChannelGroupImpl$1.run(AsynchronousChannelGroupImpl.java:112)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>   at java.lang.Thread.run(Thread.java:745)
>Locked ownable synchronizers:
>   - locked <6d02d7ed> (a java.util.concurrent.ThreadPoolExecutor$Worker)
> "sshd-SshServer[67de991c]-nio2-thread-2" - Thread t@124
>java.lang.Thread.State: TIMED_WAITING
>   at java.lang.Object.wait(Native Method)
>   - waiting on <7e9f4eff> (a java.lang.Object)
>   at 
> org.apache.sshd.client.channel.AbstractClientChannel.waitFor(AbstractClientChannel.java:244)
>   at 
> org.apache.sshd.common.forward.DefaultTcpipForwarder$StaticIoHandler.messageReceived(DefaultTcpipForwarder.java:984)
>   at 
> org.apache.sshd.common.io.nio2.Nio2Session.handleReadCycleCompletion(Nio2Session.java:276)
>   at 
> org.apache.sshd.common.io.nio2.Nio2Session$1.onCompleted(Nio2Session.java:256)
>   at 
> org.apache.sshd.common.io.nio2.Nio2Session$1.onCompleted(Nio2Session.java:253)
>   at 
> org.apache.sshd.common.io.nio2.Nio2CompletionHandler.lambda$completed$0(Nio2CompletionHandler.java:38)
>   at 
> org.apache.sshd.common.io.nio2.Nio2CompletionHandler$$Lambda$45/1071326492.run(Unknown
>  Source)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at 
> org.apache.sshd.common.io.nio2.Nio2CompletionHandler.completed(Nio2CompletionHandler.java:37)
>   at sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:126)
>   at sun.nio.ch.Invoker$2.run(Invoker.java:218)
>   at 
> sun.nio.ch.AsynchronousChannelGroupImpl$1.run(AsynchronousChannelGroupImpl.java:112)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>   at

[jira] [Commented] (SSHD-721) deadlock: all nio workers wait to be woken up

2018-04-13 Thread Bill Kuker (JIRA)


[ 
https://issues.apache.org/jira/browse/SSHD-721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16437233#comment-16437233
 ] 

Bill Kuker commented on SSHD-721:
-

I am uncertain of the answers to these. My full project includes keepalives and 
checks for crashed servers and clients and bad network connections, so after a 
certain amount of time stuck in this state the client applications kill their 
SSH clients and reconnect...

Once all clients with port forwards have killed themselves sshd begins 
operating normally.

Very similar to Markus's problem is that one web browser - web server 
connection over one port forward can get sshd into this state where all port 
forwards for all clients (not just the one client with the http port forward) 
halt.

Because at this point sshd is using no cpu, allocating no memory, executing no 
code, I'd be inclined to call it deadlock and a bug rather than a performance 
issue.

Increasing the number of NIO workers eases my pain, but my assumption would be 
that for any # of NIO workers there is a client behavior that can trigger the 
same deadlock.

I can say this problem began when I upgraded from 0.3.0 to 1.2.0 and still 
occurs in 1.6.0. I am working on updating to 1.7.0 but there are some API 
changes so that'll take a little longer. There also seems to be some time based 
component to it. It happens more for some people bouncing from toronto to paris 
to ottawa than it does for people testing it on the same lan. yay.

I guess my bottom line is that if someone from the SSHD team agrees that this 
is probably a bug I'll work real hard to reproduce it with the simplest test 
case I can... And maybe even a fix. I only have SSHD-85 to my name, but the 
dates on that ticket show a certain persistence ;)

> deadlock: all nio workers wait to be woken up
> -
>
> Key: SSHD-721
> URL: https://issues.apache.org/jira/browse/SSHD-721
> Project: MINA SSHD
>  Issue Type: Bug
>Affects Versions: 1.3.0, 1.4.0
>Reporter: Markus Rathgeb
>Priority: Major
>
> I am using sshd-core for a server machine (S) that accepts incoming 
> connections and port forwarding requests.
> There are client machines (C) that run servers that should be accessible by a 
> tunnel to the server.
> On the client machines (C) also an implementation using sshd-core is running 
> that establish the connection to the server (S) and initiate the port 
> forwarding.
> Other clients are using the tunnelled connection to communicate with the 
> servers that are running on the client machines (C).
> Sometimes I realized that no data is transferred anymore (through the 
> tunnels).
> All the worker reside in the waitFor function and no one wakes them up.
> {noformat}
> "sshd-SshServer[67de991c]-nio2-thread-3" - Thread t@125
>java.lang.Thread.State: TIMED_WAITING
>   at java.lang.Object.wait(Native Method)
>   - waiting on <132c6b60> (a java.lang.Object)
>   at 
> org.apache.sshd.client.channel.AbstractClientChannel.waitFor(AbstractClientChannel.java:244)
>   at 
> org.apache.sshd.common.forward.DefaultTcpipForwarder$StaticIoHandler.messageReceived(DefaultTcpipForwarder.java:984)
>   at 
> org.apache.sshd.common.io.nio2.Nio2Session.handleReadCycleCompletion(Nio2Session.java:276)
>   at 
> org.apache.sshd.common.io.nio2.Nio2Session$1.onCompleted(Nio2Session.java:256)
>   at 
> org.apache.sshd.common.io.nio2.Nio2Session$1.onCompleted(Nio2Session.java:253)
>   at 
> org.apache.sshd.common.io.nio2.Nio2CompletionHandler.lambda$completed$0(Nio2CompletionHandler.java:38)
>   at 
> org.apache.sshd.common.io.nio2.Nio2CompletionHandler$$Lambda$45/1071326492.run(Unknown
>  Source)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at 
> org.apache.sshd.common.io.nio2.Nio2CompletionHandler.completed(Nio2CompletionHandler.java:37)
>   at sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:126)
>   at sun.nio.ch.Invoker$2.run(Invoker.java:218)
>   at 
> sun.nio.ch.AsynchronousChannelGroupImpl$1.run(AsynchronousChannelGroupImpl.java:112)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>   at java.lang.Thread.run(Thread.java:745)
>Locked ownable synchronizers:
>   - locked <6d02d7ed> (a java.util.concurrent.ThreadPoolExecutor$Worker)
> "sshd-SshServer[67de991c]-nio2-thread-2" - Thread t@124
>java.lang.Thread.State: TIMED_WAITING
>   at java.lang.Object.wait(Native Method)
>   - waiting on <7e9f4eff> (a java.lang.Object)
>   at 
> org.apache.sshd.client.channel.AbstractClientChannel.waitFor(AbstractClientChannel.java:244)
>   at 
>

[jira] [Commented] (SSHD-721) deadlock: all nio workers wait to be woken up

2018-04-12 Thread Jonathan Valliere (JIRA)


[ 
https://issues.apache.org/jira/browse/SSHD-721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16436164#comment-16436164
 ] 

Jonathan Valliere commented on SSHD-721:


Is this a true deadlock, meaning "does it prevent the creation of new ssh 
sessions to the server?"  If you can create a new session, can you execute a 
command like "time" and eventually get a result?  I imagine that port 
forwarding might be a fully blocking operation and would accumulate in your 
thread scheduler blocking forever, or until the clients close.

> deadlock: all nio workers wait to be woken up
> -
>
> Key: SSHD-721
> URL: https://issues.apache.org/jira/browse/SSHD-721
> Project: MINA SSHD
>  Issue Type: Bug
>Affects Versions: 1.3.0, 1.4.0
>Reporter: Markus Rathgeb
>Priority: Major
>
> I am using sshd-core for a server machine (S) that accepts incoming 
> connections and port forwarding requests.
> There are client machines (C) that run servers that should be accessible by a 
> tunnel to the server.
> On the client machines (C) also an implementation using sshd-core is running 
> that establish the connection to the server (S) and initiate the port 
> forwarding.
> Other clients are using the tunnelled connection to communicate with the 
> servers that are running on the client machines (C).
> Sometimes I realized that no data is transferred anymore (through the 
> tunnels).
> All the worker reside in the waitFor function and no one wakes them up.
> {noformat}
> "sshd-SshServer[67de991c]-nio2-thread-3" - Thread t@125
>java.lang.Thread.State: TIMED_WAITING
>   at java.lang.Object.wait(Native Method)
>   - waiting on <132c6b60> (a java.lang.Object)
>   at 
> org.apache.sshd.client.channel.AbstractClientChannel.waitFor(AbstractClientChannel.java:244)
>   at 
> org.apache.sshd.common.forward.DefaultTcpipForwarder$StaticIoHandler.messageReceived(DefaultTcpipForwarder.java:984)
>   at 
> org.apache.sshd.common.io.nio2.Nio2Session.handleReadCycleCompletion(Nio2Session.java:276)
>   at 
> org.apache.sshd.common.io.nio2.Nio2Session$1.onCompleted(Nio2Session.java:256)
>   at 
> org.apache.sshd.common.io.nio2.Nio2Session$1.onCompleted(Nio2Session.java:253)
>   at 
> org.apache.sshd.common.io.nio2.Nio2CompletionHandler.lambda$completed$0(Nio2CompletionHandler.java:38)
>   at 
> org.apache.sshd.common.io.nio2.Nio2CompletionHandler$$Lambda$45/1071326492.run(Unknown
>  Source)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at 
> org.apache.sshd.common.io.nio2.Nio2CompletionHandler.completed(Nio2CompletionHandler.java:37)
>   at sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:126)
>   at sun.nio.ch.Invoker$2.run(Invoker.java:218)
>   at 
> sun.nio.ch.AsynchronousChannelGroupImpl$1.run(AsynchronousChannelGroupImpl.java:112)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>   at java.lang.Thread.run(Thread.java:745)
>Locked ownable synchronizers:
>   - locked <6d02d7ed> (a java.util.concurrent.ThreadPoolExecutor$Worker)
> "sshd-SshServer[67de991c]-nio2-thread-2" - Thread t@124
>java.lang.Thread.State: TIMED_WAITING
>   at java.lang.Object.wait(Native Method)
>   - waiting on <7e9f4eff> (a java.lang.Object)
>   at 
> org.apache.sshd.client.channel.AbstractClientChannel.waitFor(AbstractClientChannel.java:244)
>   at 
> org.apache.sshd.common.forward.DefaultTcpipForwarder$StaticIoHandler.messageReceived(DefaultTcpipForwarder.java:984)
>   at 
> org.apache.sshd.common.io.nio2.Nio2Session.handleReadCycleCompletion(Nio2Session.java:276)
>   at 
> org.apache.sshd.common.io.nio2.Nio2Session$1.onCompleted(Nio2Session.java:256)
>   at 
> org.apache.sshd.common.io.nio2.Nio2Session$1.onCompleted(Nio2Session.java:253)
>   at 
> org.apache.sshd.common.io.nio2.Nio2CompletionHandler.lambda$completed$0(Nio2CompletionHandler.java:38)
>   at 
> org.apache.sshd.common.io.nio2.Nio2CompletionHandler$$Lambda$45/1071326492.run(Unknown
>  Source)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at 
> org.apache.sshd.common.io.nio2.Nio2CompletionHandler.completed(Nio2CompletionHandler.java:37)
>   at sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:126)
>   at sun.nio.ch.Invoker$2.run(Invoker.java:218)
>   at 
> sun.nio.ch.AsynchronousChannelGroupImpl$1.run(AsynchronousChannelGroupImpl.java:112)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>   at java.lang.Thread.run(Thread.java:745)
>Locked ownable synchronizers:
>

[jira] [Commented] (SSHD-721) deadlock: all nio workers wait to be woken up

2018-04-12 Thread Bill Kuker (JIRA)


[ 
https://issues.apache.org/jira/browse/SSHD-721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16436082#comment-16436082
 ] 

Bill Kuker commented on SSHD-721:
-

I am hitting this issue too.

[~lgoldstein],
{quote}{{the number of workers is not big enough}} - the SSHD code comes with 
some reasonable defaults - if your code needs more resources, then it should 
configure them (unless I am missing something). 
{quote}
 IMHO, and I may be missing something, having too few worker threads should 
lead to poor performance, not total deadlock.

On the upside increasing my # of worker threads is at least easing my pain.

 

> deadlock: all nio workers wait to be woken up
> -
>
> Key: SSHD-721
> URL: https://issues.apache.org/jira/browse/SSHD-721
> Project: MINA SSHD
>  Issue Type: Bug
>Affects Versions: 1.3.0, 1.4.0
>Reporter: Markus Rathgeb
>Priority: Major
>
> I am using sshd-core for a server machine (S) that accepts incoming 
> connections and port forwarding requests.
> There are client machines (C) that run servers that should be accessible by a 
> tunnel to the server.
> On the client machines (C) also an implementation using sshd-core is running 
> that establish the connection to the server (S) and initiate the port 
> forwarding.
> Other clients are using the tunnelled connection to communicate with the 
> servers that are running on the client machines (C).
> Sometimes I realized that no data is transferred anymore (through the 
> tunnels).
> All the worker reside in the waitFor function and no one wakes them up.
> {noformat}
> "sshd-SshServer[67de991c]-nio2-thread-3" - Thread t@125
>java.lang.Thread.State: TIMED_WAITING
>   at java.lang.Object.wait(Native Method)
>   - waiting on <132c6b60> (a java.lang.Object)
>   at 
> org.apache.sshd.client.channel.AbstractClientChannel.waitFor(AbstractClientChannel.java:244)
>   at 
> org.apache.sshd.common.forward.DefaultTcpipForwarder$StaticIoHandler.messageReceived(DefaultTcpipForwarder.java:984)
>   at 
> org.apache.sshd.common.io.nio2.Nio2Session.handleReadCycleCompletion(Nio2Session.java:276)
>   at 
> org.apache.sshd.common.io.nio2.Nio2Session$1.onCompleted(Nio2Session.java:256)
>   at 
> org.apache.sshd.common.io.nio2.Nio2Session$1.onCompleted(Nio2Session.java:253)
>   at 
> org.apache.sshd.common.io.nio2.Nio2CompletionHandler.lambda$completed$0(Nio2CompletionHandler.java:38)
>   at 
> org.apache.sshd.common.io.nio2.Nio2CompletionHandler$$Lambda$45/1071326492.run(Unknown
>  Source)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at 
> org.apache.sshd.common.io.nio2.Nio2CompletionHandler.completed(Nio2CompletionHandler.java:37)
>   at sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:126)
>   at sun.nio.ch.Invoker$2.run(Invoker.java:218)
>   at 
> sun.nio.ch.AsynchronousChannelGroupImpl$1.run(AsynchronousChannelGroupImpl.java:112)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>   at java.lang.Thread.run(Thread.java:745)
>Locked ownable synchronizers:
>   - locked <6d02d7ed> (a java.util.concurrent.ThreadPoolExecutor$Worker)
> "sshd-SshServer[67de991c]-nio2-thread-2" - Thread t@124
>java.lang.Thread.State: TIMED_WAITING
>   at java.lang.Object.wait(Native Method)
>   - waiting on <7e9f4eff> (a java.lang.Object)
>   at 
> org.apache.sshd.client.channel.AbstractClientChannel.waitFor(AbstractClientChannel.java:244)
>   at 
> org.apache.sshd.common.forward.DefaultTcpipForwarder$StaticIoHandler.messageReceived(DefaultTcpipForwarder.java:984)
>   at 
> org.apache.sshd.common.io.nio2.Nio2Session.handleReadCycleCompletion(Nio2Session.java:276)
>   at 
> org.apache.sshd.common.io.nio2.Nio2Session$1.onCompleted(Nio2Session.java:256)
>   at 
> org.apache.sshd.common.io.nio2.Nio2Session$1.onCompleted(Nio2Session.java:253)
>   at 
> org.apache.sshd.common.io.nio2.Nio2CompletionHandler.lambda$completed$0(Nio2CompletionHandler.java:38)
>   at 
> org.apache.sshd.common.io.nio2.Nio2CompletionHandler$$Lambda$45/1071326492.run(Unknown
>  Source)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at 
> org.apache.sshd.common.io.nio2.Nio2CompletionHandler.completed(Nio2CompletionHandler.java:37)
>   at sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:126)
>   at sun.nio.ch.Invoker$2.run(Invoker.java:218)
>   at 
> sun.nio.ch.AsynchronousChannelGroupImpl$1.run(AsynchronousChannelGroupImpl.java:112)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>

[jira] [Commented] (SSHD-721) deadlock: all nio workers wait to be woken up

2016-12-11 Thread Markus Rathgeb (JIRA)


[ 
https://issues.apache.org/jira/browse/SSHD-721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15739780#comment-15739780
 ] 

Markus Rathgeb commented on SSHD-721:
-

Hi,

thank you for your reply.
It seems to me I was not able to express my problem correctly.
Let's try it again - please be patient.


the code example I posted above has been created for one and only one reason:
To give you an example to reproduce the problem.
The example is a very heavy stress test but I didn't find another way to 
reproduce it in a simple manner.

The problem I run into is, that the same number of session could be handled 
most of the time, but sometimes the server runs into a dead lock.

Also, ONE client using a forwarded port could run the server into that dead 
lock (I hope the naming is okay) situation.
If ONE client causes this scenario, the handling of this client and the 
handling of all other clients is not further proceeded.

The non-blocking I/O API is used, so the number of clients does not depend 
(directly) on the number of workers. Correct?
If I increase the number of workers multiple stuff could be handled in 
parallel. Is this a assumption correct?

If my usage of the sshd code in the example I created to demonstrate the 
problem is correct (I really don't know, perhaps I am doing something wrong), a 
change to the number of NIO workers sounds to me like changing an unit test so 
it does not fail anymore.

For me the situation looks like this one:
One HTTP client that connects to a WebServer server is able to drive the server 
in a situation, so it does not talk to any client anymore. I think we agree 
this is a bug in the server code. Don't we?

For me it would be okay, if no SSH session could be created anymore, because no 
new one could be handled.
But it looks like also all existing ones are not handled anymore.
Also there is no real escape but dropping the socket connection manually.

To summarize:
* ports are forwarded by sshd and that ones are used by some clients
* the same number of clients could be handled in parallel most of the time
* one client could use one forwarded port in way, that results in a situation 
that all workers wait for something
* the one client that communication caused that situation is not served anymore
* no other client could use any forwarded anymore
* the server could not be used by any client anymore

> deadlock: all nio workers wait to be woken up
> -
>
> Key: SSHD-721
> URL: https://issues.apache.org/jira/browse/SSHD-721
> Project: MINA SSHD
>  Issue Type: Bug
>Affects Versions: 1.3.0, 1.4.0
>Reporter: Markus Rathgeb
>
> I am using sshd-core for a server machine (S) that accepts incoming 
> connections and port forwarding requests.
> There are client machines (C) that run servers that should be accessible by a 
> tunnel to the server.
> On the client machines (C) also an implementation using sshd-core is running 
> that establish the connection to the server (S) and initiate the port 
> forwarding.
> Other clients are using the tunnelled connection to communicate with the 
> servers that are running on the client machines (C).
> Sometimes I realized that no data is transferred anymore (through the 
> tunnels).
> All the worker reside in the waitFor function and no one wakes them up.
> {noformat}
> "sshd-SshServer[67de991c]-nio2-thread-3" - Thread t@125
>java.lang.Thread.State: TIMED_WAITING
>   at java.lang.Object.wait(Native Method)
>   - waiting on <132c6b60> (a java.lang.Object)
>   at 
> org.apache.sshd.client.channel.AbstractClientChannel.waitFor(AbstractClientChannel.java:244)
>   at 
> org.apache.sshd.common.forward.DefaultTcpipForwarder$StaticIoHandler.messageReceived(DefaultTcpipForwarder.java:984)
>   at 
> org.apache.sshd.common.io.nio2.Nio2Session.handleReadCycleCompletion(Nio2Session.java:276)
>   at 
> org.apache.sshd.common.io.nio2.Nio2Session$1.onCompleted(Nio2Session.java:256)
>   at 
> org.apache.sshd.common.io.nio2.Nio2Session$1.onCompleted(Nio2Session.java:253)
>   at 
> org.apache.sshd.common.io.nio2.Nio2CompletionHandler.lambda$completed$0(Nio2CompletionHandler.java:38)
>   at 
> org.apache.sshd.common.io.nio2.Nio2CompletionHandler$$Lambda$45/1071326492.run(Unknown
>  Source)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at 
> org.apache.sshd.common.io.nio2.Nio2CompletionHandler.completed(Nio2CompletionHandler.java:37)
>   at sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:126)
>   at sun.nio.ch.Invoker$2.run(Invoker.java:218)
>   at 
> sun.nio.ch.AsynchronousChannelGroupImpl$1.run(AsynchronousChannelGroupImpl.java:112)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>   at 
>

[jira] [Commented] (SSHD-721) deadlock: all nio workers wait to be woken up

2016-12-10 Thread Goldstein Lyor (JIRA)


[ 
https://issues.apache.org/jira/browse/SSHD-721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15737563#comment-15737563
 ] 

Goldstein Lyor commented on SSHD-721:
-

I don't understand the issue - your [very own 
remark|https://github.com/maggu2810/sshd-deadlock/tree/first-report] seems to 
indicate the problem and its solution:
{quote}
As you can see if the number of workers is not big enough after some time all 
NIO workers are waiting and there is no one to wake them up.
{quote}

{{the number of workers is not big enough}} - the SSHD code comes with some 
reasonable defaults - if your code needs more resources, then it should 
configure them (unless I am missing something). As far as {{All the worker 
reside in the waitFor function}} - the code seems to be doing what it was 
intended. The NIO workers usage depends (indirectly) on the number of 
concurrent active sessions. You may have a point about maybe using this pool 
more optimally - but this is a separate issue (which you are welcome to open 
and describe *how* to manage it better - not a simple task...). Does your code 
"freeze" when you increase the configured NIO workers thread pool ? If not, 
then unless I am missing something, this does not look like a bug but rather an 
improvement issue.

> deadlock: all nio workers wait to be woken up
> -
>
> Key: SSHD-721
> URL: https://issues.apache.org/jira/browse/SSHD-721
> Project: MINA SSHD
>  Issue Type: Bug
>Affects Versions: 1.3.0, 1.4.0
>Reporter: Markus Rathgeb
>
> I am using sshd-core for a server machine (S) that accepts incoming 
> connections and port forwarding requests.
> There are client machines (C) that run servers that should be accessible by a 
> tunnel to the server.
> On the client machines (C) also an implementation using sshd-core is running 
> that establish the connection to the server (S) and initiate the port 
> forwarding.
> Other clients are using the tunnelled connection to communicate with the 
> servers that are running on the client machines (C).
> Sometimes I realized that no data is transferred anymore (through the 
> tunnels).
> All the worker reside in the waitFor function and no one wakes them up.
> {noformat}
> "sshd-SshServer[67de991c]-nio2-thread-3" - Thread t@125
>java.lang.Thread.State: TIMED_WAITING
>   at java.lang.Object.wait(Native Method)
>   - waiting on <132c6b60> (a java.lang.Object)
>   at 
> org.apache.sshd.client.channel.AbstractClientChannel.waitFor(AbstractClientChannel.java:244)
>   at 
> org.apache.sshd.common.forward.DefaultTcpipForwarder$StaticIoHandler.messageReceived(DefaultTcpipForwarder.java:984)
>   at 
> org.apache.sshd.common.io.nio2.Nio2Session.handleReadCycleCompletion(Nio2Session.java:276)
>   at 
> org.apache.sshd.common.io.nio2.Nio2Session$1.onCompleted(Nio2Session.java:256)
>   at 
> org.apache.sshd.common.io.nio2.Nio2Session$1.onCompleted(Nio2Session.java:253)
>   at 
> org.apache.sshd.common.io.nio2.Nio2CompletionHandler.lambda$completed$0(Nio2CompletionHandler.java:38)
>   at 
> org.apache.sshd.common.io.nio2.Nio2CompletionHandler$$Lambda$45/1071326492.run(Unknown
>  Source)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at 
> org.apache.sshd.common.io.nio2.Nio2CompletionHandler.completed(Nio2CompletionHandler.java:37)
>   at sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:126)
>   at sun.nio.ch.Invoker$2.run(Invoker.java:218)
>   at 
> sun.nio.ch.AsynchronousChannelGroupImpl$1.run(AsynchronousChannelGroupImpl.java:112)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>   at java.lang.Thread.run(Thread.java:745)
>Locked ownable synchronizers:
>   - locked <6d02d7ed> (a java.util.concurrent.ThreadPoolExecutor$Worker)
> "sshd-SshServer[67de991c]-nio2-thread-2" - Thread t@124
>java.lang.Thread.State: TIMED_WAITING
>   at java.lang.Object.wait(Native Method)
>   - waiting on <7e9f4eff> (a java.lang.Object)
>   at 
> org.apache.sshd.client.channel.AbstractClientChannel.waitFor(AbstractClientChannel.java:244)
>   at 
> org.apache.sshd.common.forward.DefaultTcpipForwarder$StaticIoHandler.messageReceived(DefaultTcpipForwarder.java:984)
>   at 
> org.apache.sshd.common.io.nio2.Nio2Session.handleReadCycleCompletion(Nio2Session.java:276)
>   at 
> org.apache.sshd.common.io.nio2.Nio2Session$1.onCompleted(Nio2Session.java:256)
>   at 
> org.apache.sshd.common.io.nio2.Nio2Session$1.onCompleted(Nio2Session.java:253)
>   at 
> org.apache.sshd.common.io.nio2.Nio2CompletionHandler.lambda$completed$0(Nio2CompletionHandler.java:38)
>   at 
>

[jira] [Commented] (SSHD-721) deadlock: all nio workers wait to be woken up

[jira] [Commented] (SSHD-721) deadlock: all nio workers wait to be woken up

[jira] [Commented] (SSHD-721) deadlock: all nio workers wait to be woken up

[jira] [Commented] (SSHD-721) deadlock: all nio workers wait to be woken up

[jira] [Commented] (SSHD-721) deadlock: all nio workers wait to be woken up

[jira] [Commented] (SSHD-721) deadlock: all nio workers wait to be woken up

[jira] [Commented] (SSHD-721) deadlock: all nio workers wait to be woken up

[jira] [Commented] (SSHD-721) deadlock: all nio workers wait to be woken up

[jira] [Commented] (SSHD-721) deadlock: all nio workers wait to be woken up

[jira] [Commented] (SSHD-721) deadlock: all nio workers wait to be woken up

[jira] [Commented] (SSHD-721) deadlock: all nio workers wait to be woken up

[jira] [Commented] (SSHD-721) deadlock: all nio workers wait to be woken up

[jira] [Commented] (SSHD-721) deadlock: all nio workers wait to be woken up

[jira] [Commented] (SSHD-721) deadlock: all nio workers wait to be woken up

[jira] [Commented] (SSHD-721) deadlock: all nio workers wait to be woken up

[jira] [Commented] (SSHD-721) deadlock: all nio workers wait to be woken up

[jira] [Commented] (SSHD-721) deadlock: all nio workers wait to be woken up

[jira] [Commented] (SSHD-721) deadlock: all nio workers wait to be woken up

[jira] [Commented] (SSHD-721) deadlock: all nio workers wait to be woken up

[jira] [Commented] (SSHD-721) deadlock: all nio workers wait to be woken up

20 matches

Site Navigation

Mail list logo

Footer information