[jira] [Comment Edited] (SSHD-721) deadlock: all nio workers wait to be woken up
[
https://issues.apache.org/jira/browse/SSHD-721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16438803#comment-16438803
]
Guillaume Nodet edited comment on SSHD-721 at 4/15/18 6:58 PM:
---
I think the code needs to be fixed: you can't perform a wait on the channel
from a worker thread or you risk hitting this very issue: i.e. thread pool
exhaustion.
Could you try the following patch :
{code}
diff --git
a/sshd-core/src/main/java/org/apache/sshd/common/forward/DefaultForwardingFilter.java
b/sshd-core/src/main/java/org/apache/sshd/common/forward/DefaultForwardingFilter.java
index 79364152..751f73a3 100644
---
a/sshd-core/src/main/java/org/apache/sshd/common/forward/DefaultForwardingFilter.java
+++
b/sshd-core/src/main/java/org/apache/sshd/common/forward/DefaultForwardingFilter.java
@@ -37,6 +37,7 @@ import java.util.concurrent.TimeUnit;
import java.util.concurrent.atomic.AtomicLong;
import org.apache.sshd.client.channel.ClientChannelEvent;
+import org.apache.sshd.client.future.OpenFuture;
import org.apache.sshd.common.Closeable;
import org.apache.sshd.common.Factory;
import org.apache.sshd.common.FactoryManager;
@@ -979,15 +980,22 @@ public class DefaultForwardingFilter
session, channel, totalMessages,
message.available());
}
-Collection result =
channel.waitFor(STATIC_IO_MSG_RECEIVED_EVENTS, Long.MAX_VALUE);
-if (traceEnabled) {
-log.trace("messageReceived({}) channel={}, count={}, len={}
wait result: {}",
- session, channel, totalMessages,
message.available(), result);
+OpenFuture future = channel.getOpenFuture();
+if (future.isOpened()) {
+OutputStream outputStream = channel.getInvertedIn();
+outputStream.write(buffer.array(), buffer.rpos(),
buffer.available());
+outputStream.flush();
+} else {
+future.addListener(f -> {
+try {
+OutputStream outputStream = channel.getInvertedIn();
+outputStream.write(buffer.array(), buffer.rpos(),
buffer.available());
+outputStream.flush();
+} catch (IOException e) {
+channel.getSession().exceptionCaught(e);
+}
+});
}
-
-OutputStream outputStream = channel.getInvertedIn();
-outputStream.write(buffer.array(), buffer.rpos(),
buffer.available());
-outputStream.flush();
}
@Override
{code}
was (Author: gnt):
I think the code needs to be fixed: you can't perform an infinite wait on the
channel from a worker thread or you'll run into this very issue: i.e. thread
pool exhaustion.
Could you try the following patch :
{code}
diff --git
a/sshd-core/src/main/java/org/apache/sshd/common/forward/DefaultForwardingFilter.java
b/sshd-core/src/main/java/org/apache/sshd/common/forward/DefaultForwardingFilter.java
index 79364152..751f73a3 100644
---
a/sshd-core/src/main/java/org/apache/sshd/common/forward/DefaultForwardingFilter.java
+++
b/sshd-core/src/main/java/org/apache/sshd/common/forward/DefaultForwardingFilter.java
@@ -37,6 +37,7 @@ import java.util.concurrent.TimeUnit;
import java.util.concurrent.atomic.AtomicLong;
import org.apache.sshd.client.channel.ClientChannelEvent;
+import org.apache.sshd.client.future.OpenFuture;
import org.apache.sshd.common.Closeable;
import org.apache.sshd.common.Factory;
import org.apache.sshd.common.FactoryManager;
@@ -979,15 +980,22 @@ public class DefaultForwardingFilter
session, channel, totalMessages,
message.available());
}
-Collection result =
channel.waitFor(STATIC_IO_MSG_RECEIVED_EVENTS, Long.MAX_VALUE);
-if (traceEnabled) {
-log.trace("messageReceived({}) channel={}, count={}, len={}
wait result: {}",
- session, channel, totalMessages,
message.available(), result);
+OpenFuture future = channel.getOpenFuture();
+if (future.isOpened()) {
+OutputStream outputStream = channel.getInvertedIn();
+outputStream.write(buffer.array(), buffer.rpos(),
buffer.available());
+outputStream.flush();
+} else {
+future.addListener(f -> {
+try {
+OutputStream outputStream = channel.getInvertedIn();
+outputStream.write(buffer.array(), buffer.rpos(),
buffer.available());
+outputStream.flush();
+} catch (IOException e) {
+channel.getSession().exceptionCaught(e);
+}
+});
}
-
-OutputStream
[jira] [Comment Edited] (SSHD-721) deadlock: all nio workers wait to be woken up
[ https://issues.apache.org/jira/browse/SSHD-721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16438353#comment-16438353 ] Bill Kuker edited comment on SSHD-721 at 4/14/18 2:12 PM: -- I am zeroing in on reproducing it... (In the following steps "web browser" and "web server" are just tiny java functions playing those roles) # Create an SSH Server, SSH Client and a port forward listening on the SSH Server. # Configure the SSH Server with *N* nio worker threads. # Start the "Web Server" that accepts a connection, reads some data from the client, writes some data to the client, closes the connection. # Launch *W* "Web Browsers" that each open a connection to the Web Server via the port forward listening on the SSH Server. # Use a cyclic barrier or similar to make all *W* Web Browsers connect and send their request simultaneously. (/) When *W < N* everything works fine. (x) When *W >= N* then the SSHD Server completely stops working. It will never recover (Unless some outside force kills some of the processes or tcp connections). *I will be providing a Junit test case once I have cleaned up my code.* My guess is that the SSH Server can handle N-1 port forwards connecting at once, and the 1 remaining worker thread is used for communication to the SSH Client. Once all worker threads are handling new connections to a port forward, no thread is available to make progress on communication to the sshd client. Replacing the nio2 thread pool, which is created by ThreadUtils::newFixedThreadPool() with a pool created by ThreadUtils::newCachedThreadPool() seems to fix the problem, although I am not suggesting that as the final fix. If my guess is correct, my suggestion for the fix would be different worker pools for SSH Server / SSH Client communication and for handling the port forwards. This would allow SSHD to operate when the number of simultaneous port forwarded connections being established is larger than the number of threads in either pool, and also allow users to tune these two thread pools separately depending on their usage patterns. was (Author: bkuker1): I am zeroing in on reproducing it... (In the following steps "web browser" and "web server" are just tiny java functions playing those roles) # Create an SSH Server, SSH Client and a port forward listening on the SSH Server. # Configure the SSH Server with *N* nio worker threads. # Start the "Web Server" that accepts a connection, reads some data from the client, writes some data to the client, closes the connection. # Launch *W* "Web Browsers" that each open a connection to the Web Server via the port forward listening on the SSH Server. # Use a cyclic barrier or similar to make all *W* Web Browsers connect and send their request simultaneously. (/) When *W < N* everything works fine. (x) When *W >= N* then the SSHD Server completely stops working. It will never recover (Unless some outside force kills some of the processes or tcp connections). *I will be providing a Junit test case once I have cleaned up my code.* My guess is that the SSH Server can handle N-1 port forwards connecting at once, and the 1 extra worker thread is used for communication to the SSH Client. Once all worker threads are handling new connections to a port forward, no thread is available to make progress on communication to the sshd client. Replacing the nio2 thread pool, which is created by ThreadUtils::newFixedThreadPool() with a pool created by ThreadUtils::newCachedThreadPool() seems to fix the problem, although I am not suggesting that as the final fix. If my guess is correct, my suggestion for the fix would be different worker pools for SSH Server / SSH Client communication and for handling the port forwards. This would allow SSHD to operate when the number of simultaneous port forwarded connections being established is larger than the number of threads in either pool, and also allow users to tune these two thread pools separately depending on their usage patterns. > deadlock: all nio workers wait to be woken up > - > > Key: SSHD-721 > URL: https://issues.apache.org/jira/browse/SSHD-721 > Project: MINA SSHD > Issue Type: Bug >Affects Versions: 1.3.0, 1.4.0 >Reporter: Markus Rathgeb >Priority: Major > > I am using sshd-core for a server machine (S) that accepts incoming > connections and port forwarding requests. > There are client machines (C) that run servers that should be accessible by a > tunnel to the server. > On the client machines (C) also an implementation using sshd-core is running > that establish the connection to the server (S) and initiate the port > forwarding. > Other clients are using the tunnelled connection to communicate with the > servers that are running on the client
[jira] [Comment Edited] (SSHD-721) deadlock: all nio workers wait to be woken up
[ https://issues.apache.org/jira/browse/SSHD-721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16438353#comment-16438353 ] Bill Kuker edited comment on SSHD-721 at 4/14/18 2:10 PM: -- I am zeroing in on reproducing it... (In the following steps "web browser" and "web server" are just tiny java functions playing those roles) # Create an SSH Server, SSH Client and a port forward listening on the SSH Server. # Configure the SSH Server with *N* nio worker threads. # Start the "Web Server" that accepts a connection, reads some data from the client, writes some data to the client, closes the connection. # Launch *W* "Web Browsers" that each open a connection to the Web Server via the port forward listening on the SSH Server. # Use a cyclic barrier or similar to make all *W* Web Browsers connect and send their request simultaneously. (/) When *W < N* everything works fine. (x) When *W >= N* then the SSHD Server completely stops working. It will never recover (Unless some outside force kills some of the processes or tcp connections). *I will be providing a Junit test case once I have cleaned up my code.* My guess is that the SSH Server can handle N-1 port forwards connecting at once, and the 1 extra worker thread is used for communication to the SSH Client. Once all worker threads are handling new connections to a port forward, no thread is available to make progress on communication to the sshd client. Replacing the nio2 thread pool, which is created by ThreadUtils::newFixedThreadPool() with a pool created by ThreadUtils::newCachedThreadPool() seems to fix the problem, although I am not suggesting that as the final fix. If my guess is correct, my suggestion for the fix would be different worker pools for SSH Server / SSH Client communication and for handling the port forwards. This would allow SSHD to operate when the number of simultaneous port forwarded connections being established is larger than the number of threads in either pool, and also allow users to tune these two thread pools separately depending on their usage patterns. was (Author: bkuker1): I am zeroing in on reproducing it... (In the following steps "web browser" and "web server" are just tiny java functions playing those roles) # Create an SSH Server, SSH Client and a port forward listening on the SSH Server. # Configure the SSH Server with *N* nio worker threads. # Start the "Web Server" that accepts a connection, reads some data from the client, writes some data to the client, closes the connection. # Launch *W* "Web Browsers" that each open a connection to the Web Server via the port forward listening on the SSH Server. # Use a cyclic barrier or similar to make all *W* Web Browsers connect and send their request simultaneously. (/) When *W < N* everything works fine. (x) When *W >= N* then the SSHD Server completely stops working. It will never recover (Unless some outside force kills some of the processes or tcp connections). My guess is that the SSH Server can handle N-1 port forwards connecting at once, and the 1 extra worker thread is used for communication to the SSH Client. Once all worker threads are handling new connections to a port forward, no thread is available to make progress on communication to the sshd client. I will be providing a Junit test case once I have cleaned up my code. Replacing the nio2 thread pool, which is created by ThreadUtils::newFixedThreadPool() with a pool created by ThreadUtils::newCachedThreadPool() seems to fix the problem, although I am not suggesting that as the final fix. If my guess is correct, my suggestion for the fix would be different worker pools for SSH Server / SSH Client communication and for handling the port forwards. This would allow SSHD to operate when the number of simultaneous port forwarded connections being established is larger than the number of threads in either pool, and also allow users to tune these two thread pools separately depending on their usage patterns. > deadlock: all nio workers wait to be woken up > - > > Key: SSHD-721 > URL: https://issues.apache.org/jira/browse/SSHD-721 > Project: MINA SSHD > Issue Type: Bug >Affects Versions: 1.3.0, 1.4.0 >Reporter: Markus Rathgeb >Priority: Major > > I am using sshd-core for a server machine (S) that accepts incoming > connections and port forwarding requests. > There are client machines (C) that run servers that should be accessible by a > tunnel to the server. > On the client machines (C) also an implementation using sshd-core is running > that establish the connection to the server (S) and initiate the port > forwarding. > Other clients are using the tunnelled connection to communicate with the > servers that are running on the client
[jira] [Comment Edited] (SSHD-721) deadlock: all nio workers wait to be woken up
[
https://issues.apache.org/jira/browse/SSHD-721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16437253#comment-16437253
]
Bill Kuker edited comment on SSHD-721 at 4/13/18 12:36 PM:
---
You know, as I am waking up this morning, I do remember noticing that in the
change from 0.3.0 to 1.2.0 the port forwards and the client-server
communication moved onto the same set of workers.
{{web browser < - > ssh server < - > ssh client < - > web server}}
Could it be that simple: All nio workers are working on communication between
the web browser and ssh server, but can not make progress without communication
between the ssh server and ssh client, but no nio worker is available to move
that data?
was (Author: bkuker1):
You know, as I am waking up this morning, I do remember noticing that in the
change from 0.3.0 to 1.2.0 the port forwards and the client-server
communication moved onto the same thread pool.
{{web browser < - > ssh server < - > ssh client < - > web server}}
Could it be that simple: All nio workers are working on communication between
the web browser and ssh server, but can not make progress without communication
between the ssh server and ssh client, but no nio worker is available to move
that data?
> deadlock: all nio workers wait to be woken up
> -
>
> Key: SSHD-721
> URL: https://issues.apache.org/jira/browse/SSHD-721
> Project: MINA SSHD
> Issue Type: Bug
>Affects Versions: 1.3.0, 1.4.0
>Reporter: Markus Rathgeb
>Priority: Major
>
> I am using sshd-core for a server machine (S) that accepts incoming
> connections and port forwarding requests.
> There are client machines (C) that run servers that should be accessible by a
> tunnel to the server.
> On the client machines (C) also an implementation using sshd-core is running
> that establish the connection to the server (S) and initiate the port
> forwarding.
> Other clients are using the tunnelled connection to communicate with the
> servers that are running on the client machines (C).
> Sometimes I realized that no data is transferred anymore (through the
> tunnels).
> All the worker reside in the waitFor function and no one wakes them up.
> {noformat}
> "sshd-SshServer[67de991c]-nio2-thread-3" - Thread t@125
>java.lang.Thread.State: TIMED_WAITING
> at java.lang.Object.wait(Native Method)
> - waiting on <132c6b60> (a java.lang.Object)
> at
> org.apache.sshd.client.channel.AbstractClientChannel.waitFor(AbstractClientChannel.java:244)
> at
> org.apache.sshd.common.forward.DefaultTcpipForwarder$StaticIoHandler.messageReceived(DefaultTcpipForwarder.java:984)
> at
> org.apache.sshd.common.io.nio2.Nio2Session.handleReadCycleCompletion(Nio2Session.java:276)
> at
> org.apache.sshd.common.io.nio2.Nio2Session$1.onCompleted(Nio2Session.java:256)
> at
> org.apache.sshd.common.io.nio2.Nio2Session$1.onCompleted(Nio2Session.java:253)
> at
> org.apache.sshd.common.io.nio2.Nio2CompletionHandler.lambda$completed$0(Nio2CompletionHandler.java:38)
> at
> org.apache.sshd.common.io.nio2.Nio2CompletionHandler$$Lambda$45/1071326492.run(Unknown
> Source)
> at java.security.AccessController.doPrivileged(Native Method)
> at
> org.apache.sshd.common.io.nio2.Nio2CompletionHandler.completed(Nio2CompletionHandler.java:37)
> at sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:126)
> at sun.nio.ch.Invoker$2.run(Invoker.java:218)
> at
> sun.nio.ch.AsynchronousChannelGroupImpl$1.run(AsynchronousChannelGroupImpl.java:112)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>Locked ownable synchronizers:
> - locked <6d02d7ed> (a java.util.concurrent.ThreadPoolExecutor$Worker)
> "sshd-SshServer[67de991c]-nio2-thread-2" - Thread t@124
>java.lang.Thread.State: TIMED_WAITING
> at java.lang.Object.wait(Native Method)
> - waiting on <7e9f4eff> (a java.lang.Object)
> at
> org.apache.sshd.client.channel.AbstractClientChannel.waitFor(AbstractClientChannel.java:244)
> at
> org.apache.sshd.common.forward.DefaultTcpipForwarder$StaticIoHandler.messageReceived(DefaultTcpipForwarder.java:984)
> at
> org.apache.sshd.common.io.nio2.Nio2Session.handleReadCycleCompletion(Nio2Session.java:276)
> at
> org.apache.sshd.common.io.nio2.Nio2Session$1.onCompleted(Nio2Session.java:256)
> at
> org.apache.sshd.common.io.nio2.Nio2Session$1.onCompleted(Nio2Session.java:253)
> at
> org.apache.sshd.common.io.nio2.Nio2CompletionHandler.lambda$completed$0(Nio2CompletionHandler.java:38)
> at
>
[jira] [Comment Edited] (SSHD-721) deadlock: all nio workers wait to be woken up
[
https://issues.apache.org/jira/browse/SSHD-721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16437253#comment-16437253
]
Bill Kuker edited comment on SSHD-721 at 4/13/18 12:34 PM:
---
You know, as I am waking up this morning, I do remember noticing that in the
change from 0.3.0 to 1.2.0 the port forwards and the client-server
communication moved onto the same thread pool.
{{web browser < - > ssh server < - > ssh client < - > web server}}
Could it be that simple: If all nio workers are working on sending or receiving
data to / from the web browser, no worker is available to send that data to the
ssh client?
was (Author: bkuker1):
You know, as I am waking up this morning, I do remember noticing that in the
change from 0.3.0 to 1.2.0 the port forwards and the client-server
communication moved onto the same thread pool.
{{web browser <-> ssh server <-> ssh client <-> web server}}
Could it be that simple: If all nio workers are working on sending or receiving
data to / from the web browser, no worker is available to send that data to the
ssh client?
> deadlock: all nio workers wait to be woken up
> -
>
> Key: SSHD-721
> URL: https://issues.apache.org/jira/browse/SSHD-721
> Project: MINA SSHD
> Issue Type: Bug
>Affects Versions: 1.3.0, 1.4.0
>Reporter: Markus Rathgeb
>Priority: Major
>
> I am using sshd-core for a server machine (S) that accepts incoming
> connections and port forwarding requests.
> There are client machines (C) that run servers that should be accessible by a
> tunnel to the server.
> On the client machines (C) also an implementation using sshd-core is running
> that establish the connection to the server (S) and initiate the port
> forwarding.
> Other clients are using the tunnelled connection to communicate with the
> servers that are running on the client machines (C).
> Sometimes I realized that no data is transferred anymore (through the
> tunnels).
> All the worker reside in the waitFor function and no one wakes them up.
> {noformat}
> "sshd-SshServer[67de991c]-nio2-thread-3" - Thread t@125
>java.lang.Thread.State: TIMED_WAITING
> at java.lang.Object.wait(Native Method)
> - waiting on <132c6b60> (a java.lang.Object)
> at
> org.apache.sshd.client.channel.AbstractClientChannel.waitFor(AbstractClientChannel.java:244)
> at
> org.apache.sshd.common.forward.DefaultTcpipForwarder$StaticIoHandler.messageReceived(DefaultTcpipForwarder.java:984)
> at
> org.apache.sshd.common.io.nio2.Nio2Session.handleReadCycleCompletion(Nio2Session.java:276)
> at
> org.apache.sshd.common.io.nio2.Nio2Session$1.onCompleted(Nio2Session.java:256)
> at
> org.apache.sshd.common.io.nio2.Nio2Session$1.onCompleted(Nio2Session.java:253)
> at
> org.apache.sshd.common.io.nio2.Nio2CompletionHandler.lambda$completed$0(Nio2CompletionHandler.java:38)
> at
> org.apache.sshd.common.io.nio2.Nio2CompletionHandler$$Lambda$45/1071326492.run(Unknown
> Source)
> at java.security.AccessController.doPrivileged(Native Method)
> at
> org.apache.sshd.common.io.nio2.Nio2CompletionHandler.completed(Nio2CompletionHandler.java:37)
> at sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:126)
> at sun.nio.ch.Invoker$2.run(Invoker.java:218)
> at
> sun.nio.ch.AsynchronousChannelGroupImpl$1.run(AsynchronousChannelGroupImpl.java:112)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>Locked ownable synchronizers:
> - locked <6d02d7ed> (a java.util.concurrent.ThreadPoolExecutor$Worker)
> "sshd-SshServer[67de991c]-nio2-thread-2" - Thread t@124
>java.lang.Thread.State: TIMED_WAITING
> at java.lang.Object.wait(Native Method)
> - waiting on <7e9f4eff> (a java.lang.Object)
> at
> org.apache.sshd.client.channel.AbstractClientChannel.waitFor(AbstractClientChannel.java:244)
> at
> org.apache.sshd.common.forward.DefaultTcpipForwarder$StaticIoHandler.messageReceived(DefaultTcpipForwarder.java:984)
> at
> org.apache.sshd.common.io.nio2.Nio2Session.handleReadCycleCompletion(Nio2Session.java:276)
> at
> org.apache.sshd.common.io.nio2.Nio2Session$1.onCompleted(Nio2Session.java:256)
> at
> org.apache.sshd.common.io.nio2.Nio2Session$1.onCompleted(Nio2Session.java:253)
> at
> org.apache.sshd.common.io.nio2.Nio2CompletionHandler.lambda$completed$0(Nio2CompletionHandler.java:38)
> at
> org.apache.sshd.common.io.nio2.Nio2CompletionHandler$$Lambda$45/1071326492.run(Unknown
> Source)
> at java.security.AccessController.doPrivileged(Native Method)
> at
>
[jira] [Comment Edited] (SSHD-721) deadlock: all nio workers wait to be woken up
[
https://issues.apache.org/jira/browse/SSHD-721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16436164#comment-16436164
]
Jonathan Valliere edited comment on SSHD-721 at 4/12/18 7:09 PM:
-
# does it prevent the creation of new ssh sessions to the server?
# If you can create a new session, can you execute a command like "time" and
eventually get a result?
# Can you close all of the port forwarding sessions and connect a new session
and execute the "time" command without having to restart the server.
I imagine that port forwarding might be a fully blocking operation and would
accumulate in your thread scheduler blocking forever, or until the clients
close.
was (Author: johnnyv):
Is this a true deadlock, meaning "does it prevent the creation of new ssh
sessions to the server?" If you can create a new session, can you execute a
command like "time" and eventually get a result? I imagine that port
forwarding might be a fully blocking operation and would accumulate in your
thread scheduler blocking forever, or until the clients close.
> deadlock: all nio workers wait to be woken up
> -
>
> Key: SSHD-721
> URL: https://issues.apache.org/jira/browse/SSHD-721
> Project: MINA SSHD
> Issue Type: Bug
>Affects Versions: 1.3.0, 1.4.0
>Reporter: Markus Rathgeb
>Priority: Major
>
> I am using sshd-core for a server machine (S) that accepts incoming
> connections and port forwarding requests.
> There are client machines (C) that run servers that should be accessible by a
> tunnel to the server.
> On the client machines (C) also an implementation using sshd-core is running
> that establish the connection to the server (S) and initiate the port
> forwarding.
> Other clients are using the tunnelled connection to communicate with the
> servers that are running on the client machines (C).
> Sometimes I realized that no data is transferred anymore (through the
> tunnels).
> All the worker reside in the waitFor function and no one wakes them up.
> {noformat}
> "sshd-SshServer[67de991c]-nio2-thread-3" - Thread t@125
>java.lang.Thread.State: TIMED_WAITING
> at java.lang.Object.wait(Native Method)
> - waiting on <132c6b60> (a java.lang.Object)
> at
> org.apache.sshd.client.channel.AbstractClientChannel.waitFor(AbstractClientChannel.java:244)
> at
> org.apache.sshd.common.forward.DefaultTcpipForwarder$StaticIoHandler.messageReceived(DefaultTcpipForwarder.java:984)
> at
> org.apache.sshd.common.io.nio2.Nio2Session.handleReadCycleCompletion(Nio2Session.java:276)
> at
> org.apache.sshd.common.io.nio2.Nio2Session$1.onCompleted(Nio2Session.java:256)
> at
> org.apache.sshd.common.io.nio2.Nio2Session$1.onCompleted(Nio2Session.java:253)
> at
> org.apache.sshd.common.io.nio2.Nio2CompletionHandler.lambda$completed$0(Nio2CompletionHandler.java:38)
> at
> org.apache.sshd.common.io.nio2.Nio2CompletionHandler$$Lambda$45/1071326492.run(Unknown
> Source)
> at java.security.AccessController.doPrivileged(Native Method)
> at
> org.apache.sshd.common.io.nio2.Nio2CompletionHandler.completed(Nio2CompletionHandler.java:37)
> at sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:126)
> at sun.nio.ch.Invoker$2.run(Invoker.java:218)
> at
> sun.nio.ch.AsynchronousChannelGroupImpl$1.run(AsynchronousChannelGroupImpl.java:112)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>Locked ownable synchronizers:
> - locked <6d02d7ed> (a java.util.concurrent.ThreadPoolExecutor$Worker)
> "sshd-SshServer[67de991c]-nio2-thread-2" - Thread t@124
>java.lang.Thread.State: TIMED_WAITING
> at java.lang.Object.wait(Native Method)
> - waiting on <7e9f4eff> (a java.lang.Object)
> at
> org.apache.sshd.client.channel.AbstractClientChannel.waitFor(AbstractClientChannel.java:244)
> at
> org.apache.sshd.common.forward.DefaultTcpipForwarder$StaticIoHandler.messageReceived(DefaultTcpipForwarder.java:984)
> at
> org.apache.sshd.common.io.nio2.Nio2Session.handleReadCycleCompletion(Nio2Session.java:276)
> at
> org.apache.sshd.common.io.nio2.Nio2Session$1.onCompleted(Nio2Session.java:256)
> at
> org.apache.sshd.common.io.nio2.Nio2Session$1.onCompleted(Nio2Session.java:253)
> at
> org.apache.sshd.common.io.nio2.Nio2CompletionHandler.lambda$completed$0(Nio2CompletionHandler.java:38)
> at
> org.apache.sshd.common.io.nio2.Nio2CompletionHandler$$Lambda$45/1071326492.run(Unknown
> Source)
> at java.security.AccessController.doPrivileged(Native Method)
> at
>
