[jira] [Commented] (SSHD-1069) SSHD stuck when parallel is greater than number of IO threads
[
https://issues.apache.org/jira/browse/SSHD-1069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17191819#comment-17191819
]
Lyor Goldstein commented on SSHD-1069:
--
{quote}
modifying ClientChannel to add "IoOutputStream getInvertedInIo();" might not be
a good idea.
I feel like it might introduce other problems.
{quote}
Indeed it seems not to "mesh" well with the current existing API(s). The idea
is sound though - perhaps a better implementation would be to expose a
{{writeInvertedInPacket(Buffer)}} API. In any case, once you believe you have
some code in place, please open a PR so we can review and discuss the code.
> SSHD stuck when parallel is greater than number of IO threads
> -
>
> Key: SSHD-1069
> URL: https://issues.apache.org/jira/browse/SSHD-1069
> Project: MINA SSHD
> Issue Type: Bug
>Affects Versions: 2.5.1
>Reporter: Feng Jiajie
>Priority: Major
>
> Step1. SSHD server:
> {code:java}
> import org.apache.sshd.common.FactoryManager;
> import org.apache.sshd.common.PropertyResolverUtils;
> import org.apache.sshd.common.util.security.SecurityUtils;
> import org.apache.sshd.server.SshServer;
> import org.apache.sshd.server.forward.AcceptAllForwardingFilter;
> import org.apache.sshd.server.keyprovider.SimpleGeneratorHostKeyProvider;
> import java.io.IOException;
> import java.nio.file.Paths;
> public class TestSshd2 {
> public static void main(String[] args) throws IOException,
> InterruptedException {
> SecurityUtils.setAPrioriDisabledProvider("BC", true);
> SshServer sshd = SshServer.setUpDefaultServer();
> // PropertyResolverUtils.updateProperty(sshd, FactoryManager.NIO_WORKERS,
> 32);
> sshd.setPort(12133);
> sshd.setKeyPairProvider(new
> SimpleGeneratorHostKeyProvider(Paths.get("/tmp/aa.key")));
> sshd.setPasswordAuthenticator((username, password, session) -> true);
> sshd.setForwardingFilter(AcceptAllForwardingFilter.INSTANCE);
> sshd.start();
> Thread.sleep(1);
> }
> }
> {code}
> Step2. start ssh client and iperf3 server:
> {code:java}
> ssh -o 'ExitOnForwardFailure yes' -p 12133 -f -x -N -T -R
> 0.0.0.0:15678:127.0.0.1:12345 test5@127.0.0.1
> iperf3 -s -p 12345
> {code}
> Step3. run iperf3 client:
> {code:java}
> iperf3 -c 127.0.0.1 -i 1 -t 20 -p 15678 -P 16
> {code}
> -P, --parallel # number of parallel client streams to run
> The problem can be reliably reproduced when -P is greater than the number of
> IO threads (the default is CPU cores +1).
> The problem disappears by turning up the FactoryManager.NIO_WORKERS:
> {code:java}
> PropertyResolverUtils.updateProperty(sshd, FactoryManager.NIO_WORKERS, 32);
> {code}
> jstack when SSHD is stuck:
> {code:java}
> "sshd-SshServer[467aecef](port=12133)-nio2-thread-5" #16 daemon prio=5
> os_prio=0 tid=0x7f7c70010800 nid=0x181cb in Object.wait()
> [0x7f7cb950]
>java.lang.Thread.State: TIMED_WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0x0005d058d5b0> (a
> org.apache.sshd.common.channel.Window)
> at
> org.apache.sshd.common.channel.Window.waitForCondition(Window.java:294)
> at org.apache.sshd.common.channel.Window.waitForSpace(Window.java:255)
> - locked <0x0005d058d5b0> (a
> org.apache.sshd.common.channel.Window)
> at
> org.apache.sshd.common.channel.ChannelOutputStream.flush(ChannelOutputStream.java:197)
> - locked <0x0005d058cf98> (a
> org.apache.sshd.common.channel.ChannelOutputStream)
> at
> org.apache.sshd.client.channel.ClientChannelPendingMessagesQueue.writeMessage(ClientChannelPendingMessagesQueue.java:175)
> at
> org.apache.sshd.client.channel.ClientChannelPendingMessagesQueue.handleIncomingMessage(ClientChannelPendingMessagesQueue.java:150)
> - locked <0x0005d058d098> (a java.util.LinkedList)
> at
> org.apache.sshd.common.forward.DefaultForwardingFilter$StaticIoHandler.messageReceived(DefaultForwardingFilter.java:1151)
> at
> org.apache.sshd.common.io.nio2.Nio2Session.handleReadCycleCompletion(Nio2Session.java:356)
> at
> org.apache.sshd.common.io.nio2.Nio2Session$1.onCompleted(Nio2Session.java:334)
> at
> org.apache.sshd.common.io.nio2.Nio2Session$1.onCompleted(Nio2Session.java:331)
> at
> org.apache.sshd.common.io.nio2.Nio2CompletionHandler.lambda$completed$0(Nio2CompletionHandler.java:38)
> at
> org.apache.sshd.common.io.nio2.Nio2CompletionHandler$$Lambda$35/1285889028.run(Unknown
> Source)
> at java.security.AccessController.doPrivileged(Native Method)
> at
> org.apache.sshd.common.io.nio2.Nio2CompletionHandler.completed(Nio2CompletionHandler.java:37)
> at sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:126)
> at
[jira] [Commented] (SSHD-1069) SSHD stuck when parallel is greater than number of IO threads
[
https://issues.apache.org/jira/browse/SSHD-1069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17191536#comment-17191536
]
Feng Jiajie commented on SSHD-1069:
---
Hi [~wolft], I found that the blocking issue in TcpipServerChannel has been
resolved by [Use asynchronous streams when forwarding
ports|https://github.com/apache/mina-sshd/commit/45f84aab59b2e11d72942cffe9d810e37ab64959].
So I tried to modify the TcpipClientChannel in the same way(temporary code):
[https://github.com/fengjiajie/mina-sshd/commit/a448cae1ca6b68a08b34e8d375fb56114561b144]
and it seemed to work.
There are problems with the commit
# issue SSHD-1070, there is a risk of OOM.
# modifying ClientChannel to add "IoOutputStream getInvertedInIo();" might not
be a good idea.
# I feel like it might introduce other problems.
> SSHD stuck when parallel is greater than number of IO threads
> -
>
> Key: SSHD-1069
> URL: https://issues.apache.org/jira/browse/SSHD-1069
> Project: MINA SSHD
> Issue Type: Bug
>Affects Versions: 2.5.1
>Reporter: Feng Jiajie
>Priority: Major
>
> Step1. SSHD server:
> {code:java}
> import org.apache.sshd.common.FactoryManager;
> import org.apache.sshd.common.PropertyResolverUtils;
> import org.apache.sshd.common.util.security.SecurityUtils;
> import org.apache.sshd.server.SshServer;
> import org.apache.sshd.server.forward.AcceptAllForwardingFilter;
> import org.apache.sshd.server.keyprovider.SimpleGeneratorHostKeyProvider;
> import java.io.IOException;
> import java.nio.file.Paths;
> public class TestSshd2 {
> public static void main(String[] args) throws IOException,
> InterruptedException {
> SecurityUtils.setAPrioriDisabledProvider("BC", true);
> SshServer sshd = SshServer.setUpDefaultServer();
> // PropertyResolverUtils.updateProperty(sshd, FactoryManager.NIO_WORKERS,
> 32);
> sshd.setPort(12133);
> sshd.setKeyPairProvider(new
> SimpleGeneratorHostKeyProvider(Paths.get("/tmp/aa.key")));
> sshd.setPasswordAuthenticator((username, password, session) -> true);
> sshd.setForwardingFilter(AcceptAllForwardingFilter.INSTANCE);
> sshd.start();
> Thread.sleep(1);
> }
> }
> {code}
> Step2. start ssh client and iperf3 server:
> {code:java}
> ssh -o 'ExitOnForwardFailure yes' -p 12133 -f -x -N -T -R
> 0.0.0.0:15678:127.0.0.1:12345 test5@127.0.0.1
> iperf3 -s -p 12345
> {code}
> Step3. run iperf3 client:
> {code:java}
> iperf3 -c 127.0.0.1 -i 1 -t 20 -p 15678 -P 16
> {code}
> -P, --parallel # number of parallel client streams to run
> The problem can be reliably reproduced when -P is greater than the number of
> IO threads (the default is CPU cores +1).
> The problem disappears by turning up the FactoryManager.NIO_WORKERS:
> {code:java}
> PropertyResolverUtils.updateProperty(sshd, FactoryManager.NIO_WORKERS, 32);
> {code}
> jstack when SSHD is stuck:
> {code:java}
> "sshd-SshServer[467aecef](port=12133)-nio2-thread-5" #16 daemon prio=5
> os_prio=0 tid=0x7f7c70010800 nid=0x181cb in Object.wait()
> [0x7f7cb950]
>java.lang.Thread.State: TIMED_WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0x0005d058d5b0> (a
> org.apache.sshd.common.channel.Window)
> at
> org.apache.sshd.common.channel.Window.waitForCondition(Window.java:294)
> at org.apache.sshd.common.channel.Window.waitForSpace(Window.java:255)
> - locked <0x0005d058d5b0> (a
> org.apache.sshd.common.channel.Window)
> at
> org.apache.sshd.common.channel.ChannelOutputStream.flush(ChannelOutputStream.java:197)
> - locked <0x0005d058cf98> (a
> org.apache.sshd.common.channel.ChannelOutputStream)
> at
> org.apache.sshd.client.channel.ClientChannelPendingMessagesQueue.writeMessage(ClientChannelPendingMessagesQueue.java:175)
> at
> org.apache.sshd.client.channel.ClientChannelPendingMessagesQueue.handleIncomingMessage(ClientChannelPendingMessagesQueue.java:150)
> - locked <0x0005d058d098> (a java.util.LinkedList)
> at
> org.apache.sshd.common.forward.DefaultForwardingFilter$StaticIoHandler.messageReceived(DefaultForwardingFilter.java:1151)
> at
> org.apache.sshd.common.io.nio2.Nio2Session.handleReadCycleCompletion(Nio2Session.java:356)
> at
> org.apache.sshd.common.io.nio2.Nio2Session$1.onCompleted(Nio2Session.java:334)
> at
> org.apache.sshd.common.io.nio2.Nio2Session$1.onCompleted(Nio2Session.java:331)
> at
> org.apache.sshd.common.io.nio2.Nio2CompletionHandler.lambda$completed$0(Nio2CompletionHandler.java:38)
> at
> org.apache.sshd.common.io.nio2.Nio2CompletionHandler$$Lambda$35/1285889028.run(Unknown
> Source)
> at java.security.AccessController.doPrivileged(Native Method)
> at
>
[jira] [Commented] (SSHD-1069) SSHD stuck when parallel is greater than number of IO threads
[
https://issues.apache.org/jira/browse/SSHD-1069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17191498#comment-17191498
]
Thomas Wolf commented on SSHD-1069:
---
Using a cached thread pool instead of the fixed thread pool provided by the
{{Nio2ServiceFactory}} resolves this. But is that the correct fix? I notice
that the {{MinaServiceFactory}} does use a cached thread pool, too.
> SSHD stuck when parallel is greater than number of IO threads
> -
>
> Key: SSHD-1069
> URL: https://issues.apache.org/jira/browse/SSHD-1069
> Project: MINA SSHD
> Issue Type: Bug
>Affects Versions: 2.5.1
>Reporter: Feng Jiajie
>Priority: Major
>
> Step1. SSHD server:
> {code:java}
> import org.apache.sshd.common.FactoryManager;
> import org.apache.sshd.common.PropertyResolverUtils;
> import org.apache.sshd.common.util.security.SecurityUtils;
> import org.apache.sshd.server.SshServer;
> import org.apache.sshd.server.forward.AcceptAllForwardingFilter;
> import org.apache.sshd.server.keyprovider.SimpleGeneratorHostKeyProvider;
> import java.io.IOException;
> import java.nio.file.Paths;
> public class TestSshd2 {
> public static void main(String[] args) throws IOException,
> InterruptedException {
> SecurityUtils.setAPrioriDisabledProvider("BC", true);
> SshServer sshd = SshServer.setUpDefaultServer();
> // PropertyResolverUtils.updateProperty(sshd, FactoryManager.NIO_WORKERS,
> 32);
> sshd.setPort(12133);
> sshd.setKeyPairProvider(new
> SimpleGeneratorHostKeyProvider(Paths.get("/tmp/aa.key")));
> sshd.setPasswordAuthenticator((username, password, session) -> true);
> sshd.setForwardingFilter(AcceptAllForwardingFilter.INSTANCE);
> sshd.start();
> Thread.sleep(1);
> }
> }
> {code}
> Step2. start ssh client and iperf3 server:
> {code:java}
> ssh -o 'ExitOnForwardFailure yes' -p 12133 -f -x -N -T -R
> 0.0.0.0:15678:127.0.0.1:12345 test5@127.0.0.1
> iperf3 -s -p 12345
> {code}
> Step3. run iperf3 client:
> {code:java}
> iperf3 -c 127.0.0.1 -i 1 -t 20 -p 15678 -P 16
> {code}
> -P, --parallel # number of parallel client streams to run
> The problem can be reliably reproduced when -P is greater than the number of
> IO threads (the default is CPU cores +1).
> The problem disappears by turning up the FactoryManager.NIO_WORKERS:
> {code:java}
> PropertyResolverUtils.updateProperty(sshd, FactoryManager.NIO_WORKERS, 32);
> {code}
> jstack when SSHD is stuck:
> {code:java}
> "sshd-SshServer[467aecef](port=12133)-nio2-thread-5" #16 daemon prio=5
> os_prio=0 tid=0x7f7c70010800 nid=0x181cb in Object.wait()
> [0x7f7cb950]
>java.lang.Thread.State: TIMED_WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0x0005d058d5b0> (a
> org.apache.sshd.common.channel.Window)
> at
> org.apache.sshd.common.channel.Window.waitForCondition(Window.java:294)
> at org.apache.sshd.common.channel.Window.waitForSpace(Window.java:255)
> - locked <0x0005d058d5b0> (a
> org.apache.sshd.common.channel.Window)
> at
> org.apache.sshd.common.channel.ChannelOutputStream.flush(ChannelOutputStream.java:197)
> - locked <0x0005d058cf98> (a
> org.apache.sshd.common.channel.ChannelOutputStream)
> at
> org.apache.sshd.client.channel.ClientChannelPendingMessagesQueue.writeMessage(ClientChannelPendingMessagesQueue.java:175)
> at
> org.apache.sshd.client.channel.ClientChannelPendingMessagesQueue.handleIncomingMessage(ClientChannelPendingMessagesQueue.java:150)
> - locked <0x0005d058d098> (a java.util.LinkedList)
> at
> org.apache.sshd.common.forward.DefaultForwardingFilter$StaticIoHandler.messageReceived(DefaultForwardingFilter.java:1151)
> at
> org.apache.sshd.common.io.nio2.Nio2Session.handleReadCycleCompletion(Nio2Session.java:356)
> at
> org.apache.sshd.common.io.nio2.Nio2Session$1.onCompleted(Nio2Session.java:334)
> at
> org.apache.sshd.common.io.nio2.Nio2Session$1.onCompleted(Nio2Session.java:331)
> at
> org.apache.sshd.common.io.nio2.Nio2CompletionHandler.lambda$completed$0(Nio2CompletionHandler.java:38)
> at
> org.apache.sshd.common.io.nio2.Nio2CompletionHandler$$Lambda$35/1285889028.run(Unknown
> Source)
> at java.security.AccessController.doPrivileged(Native Method)
> at
> org.apache.sshd.common.io.nio2.Nio2CompletionHandler.completed(Nio2CompletionHandler.java:37)
> at sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:126)
> at sun.nio.ch.Invoker.invokeDirect(Invoker.java:157)
> at
> sun.nio.ch.UnixAsynchronousSocketChannelImpl.implRead(UnixAsynchronousSocketChannelImpl.java:553)
> at
>
[jira] [Commented] (SSHD-1069) SSHD stuck when parallel is greater than number of IO threads
[
https://issues.apache.org/jira/browse/SSHD-1069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17191476#comment-17191476
]
Thomas Wolf commented on SSHD-1069:
---
Are _all_ NIO threads blocked on {{waitForSpace()}}? If so, it looks like the
readers starve writers. Which thread would eventually request the window adjust?
> SSHD stuck when parallel is greater than number of IO threads
> -
>
> Key: SSHD-1069
> URL: https://issues.apache.org/jira/browse/SSHD-1069
> Project: MINA SSHD
> Issue Type: Bug
>Affects Versions: 2.5.1
>Reporter: Feng Jiajie
>Priority: Major
>
> Step1. SSHD server:
> {code:java}
> import org.apache.sshd.common.FactoryManager;
> import org.apache.sshd.common.PropertyResolverUtils;
> import org.apache.sshd.common.util.security.SecurityUtils;
> import org.apache.sshd.server.SshServer;
> import org.apache.sshd.server.forward.AcceptAllForwardingFilter;
> import org.apache.sshd.server.keyprovider.SimpleGeneratorHostKeyProvider;
> import java.io.IOException;
> import java.nio.file.Paths;
> public class TestSshd2 {
> public static void main(String[] args) throws IOException,
> InterruptedException {
> SecurityUtils.setAPrioriDisabledProvider("BC", true);
> SshServer sshd = SshServer.setUpDefaultServer();
> // PropertyResolverUtils.updateProperty(sshd, FactoryManager.NIO_WORKERS,
> 32);
> sshd.setPort(12133);
> sshd.setKeyPairProvider(new
> SimpleGeneratorHostKeyProvider(Paths.get("/tmp/aa.key")));
> sshd.setPasswordAuthenticator((username, password, session) -> true);
> sshd.setForwardingFilter(AcceptAllForwardingFilter.INSTANCE);
> sshd.start();
> Thread.sleep(1);
> }
> }
> {code}
> Step2. start ssh client and iperf3 server:
> {code:java}
> ssh -o 'ExitOnForwardFailure yes' -p 12133 -f -x -N -T -R
> 0.0.0.0:15678:127.0.0.1:12345 test5@127.0.0.1
> iperf3 -s -p 12345
> {code}
> Step3. run iperf3 client:
> {code:java}
> iperf3 -c 127.0.0.1 -i 1 -t 20 -p 15678 -P 16
> {code}
> -P, --parallel # number of parallel client streams to run
> The problem can be reliably reproduced when -P is greater than the number of
> IO threads (the default is CPU cores +1).
> The problem disappears by turning up the FactoryManager.NIO_WORKERS:
> {code:java}
> PropertyResolverUtils.updateProperty(sshd, FactoryManager.NIO_WORKERS, 32);
> {code}
> jstack when SSHD is stuck:
> {code:java}
> "sshd-SshServer[467aecef](port=12133)-nio2-thread-5" #16 daemon prio=5
> os_prio=0 tid=0x7f7c70010800 nid=0x181cb in Object.wait()
> [0x7f7cb950]
>java.lang.Thread.State: TIMED_WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0x0005d058d5b0> (a
> org.apache.sshd.common.channel.Window)
> at
> org.apache.sshd.common.channel.Window.waitForCondition(Window.java:294)
> at org.apache.sshd.common.channel.Window.waitForSpace(Window.java:255)
> - locked <0x0005d058d5b0> (a
> org.apache.sshd.common.channel.Window)
> at
> org.apache.sshd.common.channel.ChannelOutputStream.flush(ChannelOutputStream.java:197)
> - locked <0x0005d058cf98> (a
> org.apache.sshd.common.channel.ChannelOutputStream)
> at
> org.apache.sshd.client.channel.ClientChannelPendingMessagesQueue.writeMessage(ClientChannelPendingMessagesQueue.java:175)
> at
> org.apache.sshd.client.channel.ClientChannelPendingMessagesQueue.handleIncomingMessage(ClientChannelPendingMessagesQueue.java:150)
> - locked <0x0005d058d098> (a java.util.LinkedList)
> at
> org.apache.sshd.common.forward.DefaultForwardingFilter$StaticIoHandler.messageReceived(DefaultForwardingFilter.java:1151)
> at
> org.apache.sshd.common.io.nio2.Nio2Session.handleReadCycleCompletion(Nio2Session.java:356)
> at
> org.apache.sshd.common.io.nio2.Nio2Session$1.onCompleted(Nio2Session.java:334)
> at
> org.apache.sshd.common.io.nio2.Nio2Session$1.onCompleted(Nio2Session.java:331)
> at
> org.apache.sshd.common.io.nio2.Nio2CompletionHandler.lambda$completed$0(Nio2CompletionHandler.java:38)
> at
> org.apache.sshd.common.io.nio2.Nio2CompletionHandler$$Lambda$35/1285889028.run(Unknown
> Source)
> at java.security.AccessController.doPrivileged(Native Method)
> at
> org.apache.sshd.common.io.nio2.Nio2CompletionHandler.completed(Nio2CompletionHandler.java:37)
> at sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:126)
> at sun.nio.ch.Invoker.invokeDirect(Invoker.java:157)
> at
> sun.nio.ch.UnixAsynchronousSocketChannelImpl.implRead(UnixAsynchronousSocketChannelImpl.java:553)
> at
> sun.nio.ch.AsynchronousSocketChannelImpl.read(AsynchronousSocketChannelImpl.java:276)
> at
>
[jira] [Commented] (SSHD-1069) SSHD stuck when parallel is greater than number of IO threads
[
https://issues.apache.org/jira/browse/SSHD-1069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17188118#comment-17188118
]
Feng Jiajie commented on SSHD-1069:
---
Hi [~gnodet], sorry to bother you, please help to see if this is a related
issue with https://issues.apache.org/jira/browse/SSHD-1003 and
https://issues.apache.org/jira/browse/SSHD-557
> SSHD stuck when parallel is greater than number of IO threads
> -
>
> Key: SSHD-1069
> URL: https://issues.apache.org/jira/browse/SSHD-1069
> Project: MINA SSHD
> Issue Type: Bug
>Affects Versions: 2.5.1
>Reporter: Feng Jiajie
>Priority: Major
>
> Step1. SSHD server:
> {code:java}
> import org.apache.sshd.common.FactoryManager;
> import org.apache.sshd.common.PropertyResolverUtils;
> import org.apache.sshd.common.util.security.SecurityUtils;
> import org.apache.sshd.server.SshServer;
> import org.apache.sshd.server.forward.AcceptAllForwardingFilter;
> import org.apache.sshd.server.keyprovider.SimpleGeneratorHostKeyProvider;
> import java.io.IOException;
> import java.nio.file.Paths;
> public class TestSshd2 {
> public static void main(String[] args) throws IOException,
> InterruptedException {
> SecurityUtils.setAPrioriDisabledProvider("BC", true);
> SshServer sshd = SshServer.setUpDefaultServer();
> // PropertyResolverUtils.updateProperty(sshd, FactoryManager.NIO_WORKERS,
> 32);
> sshd.setPort(12133);
> sshd.setKeyPairProvider(new
> SimpleGeneratorHostKeyProvider(Paths.get("/tmp/aa.key")));
> sshd.setPasswordAuthenticator((username, password, session) -> true);
> sshd.setForwardingFilter(AcceptAllForwardingFilter.INSTANCE);
> sshd.start();
> Thread.sleep(1);
> }
> }
> {code}
> Step2. start ssh client and iperf3 server:
> {code:java}
> ssh -o 'ExitOnForwardFailure yes' -p 12133 -f -x -N -T -R
> 0.0.0.0:15678:127.0.0.1:12345 test5@127.0.0.1
> iperf3 -s -p 12345
> {code}
> Step3. run iperf3 client:
> {code:java}
> iperf3 -c 127.0.0.1 -i 1 -t 20 -p 15678 -P 16
> {code}
> -P, --parallel # number of parallel client streams to run
> The problem can be reliably reproduced when -P is greater than the number of
> IO threads (the default is CPU cores +1).
> The problem disappears by turning up the FactoryManager.NIO_WORKERS:
> {code:java}
> PropertyResolverUtils.updateProperty(sshd, FactoryManager.NIO_WORKERS, 32);
> {code}
> jstack when SSHD is stuck:
> {code:java}
> "sshd-SshServer[467aecef](port=12133)-nio2-thread-5" #16 daemon prio=5
> os_prio=0 tid=0x7f7c70010800 nid=0x181cb in Object.wait()
> [0x7f7cb950]
>java.lang.Thread.State: TIMED_WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0x0005d058d5b0> (a
> org.apache.sshd.common.channel.Window)
> at
> org.apache.sshd.common.channel.Window.waitForCondition(Window.java:294)
> at org.apache.sshd.common.channel.Window.waitForSpace(Window.java:255)
> - locked <0x0005d058d5b0> (a
> org.apache.sshd.common.channel.Window)
> at
> org.apache.sshd.common.channel.ChannelOutputStream.flush(ChannelOutputStream.java:197)
> - locked <0x0005d058cf98> (a
> org.apache.sshd.common.channel.ChannelOutputStream)
> at
> org.apache.sshd.client.channel.ClientChannelPendingMessagesQueue.writeMessage(ClientChannelPendingMessagesQueue.java:175)
> at
> org.apache.sshd.client.channel.ClientChannelPendingMessagesQueue.handleIncomingMessage(ClientChannelPendingMessagesQueue.java:150)
> - locked <0x0005d058d098> (a java.util.LinkedList)
> at
> org.apache.sshd.common.forward.DefaultForwardingFilter$StaticIoHandler.messageReceived(DefaultForwardingFilter.java:1151)
> at
> org.apache.sshd.common.io.nio2.Nio2Session.handleReadCycleCompletion(Nio2Session.java:356)
> at
> org.apache.sshd.common.io.nio2.Nio2Session$1.onCompleted(Nio2Session.java:334)
> at
> org.apache.sshd.common.io.nio2.Nio2Session$1.onCompleted(Nio2Session.java:331)
> at
> org.apache.sshd.common.io.nio2.Nio2CompletionHandler.lambda$completed$0(Nio2CompletionHandler.java:38)
> at
> org.apache.sshd.common.io.nio2.Nio2CompletionHandler$$Lambda$35/1285889028.run(Unknown
> Source)
> at java.security.AccessController.doPrivileged(Native Method)
> at
> org.apache.sshd.common.io.nio2.Nio2CompletionHandler.completed(Nio2CompletionHandler.java:37)
> at sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:126)
> at sun.nio.ch.Invoker.invokeDirect(Invoker.java:157)
> at
> sun.nio.ch.UnixAsynchronousSocketChannelImpl.implRead(UnixAsynchronousSocketChannelImpl.java:553)
> at
> sun.nio.ch.AsynchronousSocketChannelImpl.read(AsynchronousSocketChannelImpl.java:276)
> at
>
