[
https://issues.apache.org/jira/browse/SSHD-1070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17192639#comment-17192639
]
Thomas Wolf edited comment on SSHD-1070 at 9/9/20, 5:53 AM:
------------------------------------------------------------
{quote}I think your implementation PR #163 has the minimal overall impact and
is more suitable for a quick fix to the problem.
{quote}
I'm not sure. I actually closed/abandoned that PR after having tried your
change. I did observe that with my change, I consistently did run into
SSHD-1069 with the iperf3 test you suggested here (with -P 8; my machine has 4
cores and thus 5 NIO threads), but with yours, it just worked fine even with -P
8, and the throughputs for -P 1 and -P 2 were identical for both approaches.
Also, if I understand your change right, it is specific for the forwarding
case, whereas mine has the risk of affecting other uses of the
{{BufferedIoOutputStream}}. But forwarding should be the only scenario where a
server would encounter such speed differences between the two ends. (Or are
there other cases?) So probably your approach is less risky.
was (Author: wolft):
{quote}
I think your implementation PR #163 has the minimal overall impact and is more
suitable for a quick fix to the problem.
{quote}
I'm not sure. I actually closed/abandoned that PR after having tried your
change. I did observe that with my change, I consistently did run into into
SSHD-1069 with the iperf3 test you suggested here (with -P 8; my machine has 4
cores and thus 5 NIO threads), but with yours, it just worked fine even with -P
8, and the throughputs for -P 1 and -P 2 were identical for both approaches.
Also, if I understand your change right, it is specific for the forwarding
case, whereas mine has the risk of affecting other uses of the
{{BufferedIoOutputstream}}. But forwarding should be the only scenario where a
server would encounter such speed differences between the two ends. (Or are
there other cases?) So probably your approach is less risky.
> OutOfMemoryError when use port forwarding
> -----------------------------------------
>
> Key: SSHD-1070
> URL: https://issues.apache.org/jira/browse/SSHD-1070
> Project: MINA SSHD
> Issue Type: Bug
> Affects Versions: 2.5.1
> Reporter: Feng Jiajie
> Priority: Major
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> Hi [~gnodet], I found this commit implemented asynchronous port forwarding:
> [https://github.com/apache/mina-sshd/commit/45f84aab59b2e11d72942cffe9d810e37ab64959#diff-33823b8546f71d77bb1d653358ecde70]
> However, when a large amount of data is returned from upstream application,
> it is possible to cause an OOM in SSHD.
> Step1. SSHD server:
> {code:java}
> import org.apache.sshd.common.FactoryManager;
> import org.apache.sshd.common.PropertyResolverUtils;
> import org.apache.sshd.common.util.security.SecurityUtils;
> import org.apache.sshd.server.SshServer;
> import org.apache.sshd.server.forward.AcceptAllForwardingFilter;
> import org.apache.sshd.server.keyprovider.SimpleGeneratorHostKeyProvider;
> import java.io.IOException;
> import java.nio.file.Paths;
> public class TestSshd2 {
> public static void main(String[] args) throws IOException,
> InterruptedException {
> SecurityUtils.setAPrioriDisabledProvider("BC", true);
> SshServer sshd = SshServer.setUpDefaultServer();
> sshd.setPort(12133);
> sshd.setKeyPairProvider(new
> SimpleGeneratorHostKeyProvider(Paths.get("/tmp/aa.key")));
> sshd.setPasswordAuthenticator((username, password, session) -> true);
> sshd.setForwardingFilter(AcceptAllForwardingFilter.INSTANCE);
> sshd.start();
> Thread.sleep(100000000);
> }
> }
> {code}
> Step2. start ssh client and iperf3 server:
> {code:java}
> ssh -o 'ExitOnForwardFailure yes' -p 12133 -f -x -N -T -L
> 0.0.0.0:15678:127.0.0.1:12345 [email protected]
> iperf3 -s -p 12345
> {code}
> Step3. run iperf3 client:
> {code:java}
> iperf3 -c 127.0.0.1 -i 1 -t 120 -p 15678 -P 8 --reverse
> {code}
> *-R, --reverse run in reverse mode (server sends, client receives)*
> SSHD will receive a lot of data but will not be able to forward it in time.
> log when OOM:
> {code:java}
> 17:52:25.195 [sshd-SshServer[5bcea91b](port=12133)-nio2-thread-1] WARN
> org.apache.sshd.common.io.nio2.Nio2Session -
> exceptionCaught(Nio2Session[local=/127.0.0.1:33524, remote=/127.0.0.1:12345])
> Exception handler threw OutOfMemoryError, closing the session: GC overhead
> limit exceeded17:52:25.195
> [sshd-SshServer[5bcea91b](port=12133)-nio2-thread-1] WARN
> org.apache.sshd.common.io.nio2.Nio2Session -
> exceptionCaught(Nio2Session[local=/127.0.0.1:33524, remote=/127.0.0.1:12345])
> Exception handler threw OutOfMemoryError, closing the session: GC overhead
> limit exceeded at
> org.apache.sshd.common.io.nio2.Nio2CompletionHandler.completed(Nio2CompletionHandler.java:37)
> at sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:126) at
> sun.nio.ch.Invoker.invokeDirect(Invoker.java:157) at
> sun.nio.ch.UnixAsynchronousSocketChannelImpl.implWrite(UnixAsynchronousSocketChannelImpl.java:736)
> at
> sun.nio.ch.AsynchronousSocketChannelImpl.write(AsynchronousSocketChannelImpl.java:382)
> at
> sun.nio.ch.AsynchronousSocketChannelImpl.write(AsynchronousSocketChannelImpl.java:399)
> at
> org.apache.sshd.common.io.nio2.Nio2Session.doWriteCycle(Nio2Session.java:420)
> at
> org.apache.sshd.common.io.nio2.Nio2Session.startWriting(Nio2Session.java:404)
> at
> org.apache.sshd.common.io.nio2.Nio2Session.finishWrite(Nio2Session.java:495)
> at
> org.apache.sshd.common.io.nio2.Nio2Session.handleCompletedWriteCycle(Nio2Session.java:465)
> at
> org.apache.sshd.common.io.nio2.Nio2Session$2.onCompleted(Nio2Session.java:429)
> at
> org.apache.sshd.common.io.nio2.Nio2Session$2.onCompleted(Nio2Session.java:426)17:52:25.639
> [sshd-SshServer[5bcea91b](port=12133)-nio2-thread-6] WARN
> org.apache.sshd.common.io.nio2.Nio2Session -
> exceptionCaught(Nio2Session[local=/127.0.0.1:33530, remote=/127.0.0.1:12345])
> Exception handler threw OutOfMemoryError, closing the session: GC overhead
> limit exceeded at
> org.apache.sshd.common.io.nio2.Nio2CompletionHandler.lambda$completed$0(Nio2CompletionHandler.java:38)
> at java.security.AccessController.doPrivileged(Native Method) at
> org.apache.sshd.common.io.nio2.Nio2CompletionHandler.completed(Nio2CompletionHandler.java:37)
> at sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:126) at
> sun.nio.ch.Invoker.invokeDirect(Invoker.java:157) at
> sun.nio.ch.UnixAsynchronousSocketChannelImpl.implWrite(UnixAsynchronousSocketChannelImpl.java:736)17:52:26.045
> [sshd-SshServer[5bcea91b](port=12133)-nio2-thread-2] DEBUG
> org.apache.sshd.common.io.nio2.Nio2Session -
> exceptionCaught(Nio2Session[local=/127.0.0.1:33522, remote=/127.0.0.1:12345])
> caught OutOfMemoryError[GC overhead limit exceeded] - calling
> handler17:52:26.045 [sshd-SshServer[5bcea91b](port=12133)-nio2-thread-2]
> DEBUG org.apache.sshd.server.forward.TcpipServerChannel -
> exceptionCaught(TcpipServerChannel[id=3,
> recipient=4]-ServerSessionImpl[test5@/127.0.0.1:53702]) signal close
> immediately=false due to OutOfMemoryError[GC overhead limit
> exceeded]17:52:26.045 [sshd-SshServer[5bcea91b](port=12133)-nio2-thread-2]
> DEBUG org.apache.sshd.server.forward.TcpipServerChannel -
> close(TcpipServerChannel[id=3,
> recipient=4]-ServerSessionImpl[test5@/127.0.0.1:53702])[Graceful] state
> already Graceful17:52:26.794
> [sshd-SshServer[5bcea91b](port=12133)-nio2-thread-3] WARN
> org.apache.sshd.common.io.nio2.Nio2Session -
> exceptionCaught(Nio2Session[local=/127.0.0.1:33528, remote=/127.0.0.1:12345])
> Exception handler threw OutOfMemoryError, closing the session: GC overhead
> limit exceeded at
> sun.nio.ch.AsynchronousSocketChannelImpl.write(AsynchronousSocketChannelImpl.java:382)
> at
> sun.nio.ch.AsynchronousSocketChannelImpl.write(AsynchronousSocketChannelImpl.java:399)
> at
> org.apache.sshd.common.io.nio2.Nio2Session.doWriteCycle(Nio2Session.java:420)
> at
> org.apache.sshd.common.io.nio2.Nio2Session.startWriting(Nio2Session.java:404)
> at
> org.apache.sshd.common.io.nio2.Nio2Session.finishWrite(Nio2Session.java:495)
> at
> org.apache.sshd.common.io.nio2.Nio2Session.handleCompletedWriteCycle(Nio2Session.java:465)
> at
> org.apache.sshd.common.io.nio2.Nio2Session$2.onCompleted(Nio2Session.java:429)
> at
> org.apache.sshd.common.io.nio2.Nio2Session$2.onCompleted(Nio2Session.java:426)
> at
> org.apache.sshd.common.io.nio2.Nio2CompletionHandler.lambda$completed$0(Nio2CompletionHandler.java:38)17:52:26.045
> [sshd-SshServer[5bcea91b](port=12133)-nio2-thread-2] DEBUG
> org.apache.sshd.common.io.nio2.Nio2Session -
> close(Nio2Session[local=/127.0.0.1:33522, remote=/127.0.0.1:12345]) Closing
> immediately at java.security.AccessController.doPrivileged(Native
> Method)17:52:26.869 [sshd-SshServer[5bcea91b](port=12133)-nio2-thread-2]
> DEBUG org.apache.sshd.common.io.nio2.Nio2Session -
> doCloseImmediately(Nio2Session[local=/127.0.0.1:33522,
> remote=/127.0.0.1:12345]) closing
> socket=sun.nio.ch.UnixAsynchronousSocketChannelImpl[connected
> local=/127.0.0.1:33522 remote=/127.0.0.1:12345] at
> org.apache.sshd.common.io.nio2.Nio2CompletionHandler.completed(Nio2CompletionHandler.java:37)
> at sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:126) at
> sun.nio.ch.Invoker.invokeDirect(Invoker.java:157)17:52:27.568
> [sshd-SshServer[5bcea91b](port=12133)-nio2-thread-6] DEBUG
> org.apache.sshd.common.io.nio2.Nio2Session -
> exceptionCaught(Nio2Session[local=/127.0.0.1:33530, remote=/127.0.0.1:12345])
> caught OutOfMemoryError[GC overhead limit exceeded] - calling
> handler17:52:27.568 [sshd-SshServer[5bcea91b](port=12133)-nio2-thread-6]
> DEBUG org.apache.sshd.server.forward.TcpipServerChannel -
> exceptionCaught(TcpipServerChannel[id=7,
> recipient=8]-ServerSessionImpl[test5@/127.0.0.1:53702]) signal close
> immediately=false due to OutOfMemoryError[GC overhead limit
> exceeded]17:52:27.568 [sshd-SshServer[5bcea91b](port=12133)-nio2-thread-6]
> DEBUG org.apache.sshd.server.forward.TcpipServerChannel -
> close(TcpipServerChannel[id=7,
> recipient=8]-ServerSessionImpl[test5@/127.0.0.1:53702])[Graceful] state
> already Graceful at
> sun.nio.ch.UnixAsynchronousSocketChannelImpl.implWrite(UnixAsynchronousSocketChannelImpl.java:736)
> at
> sun.nio.ch.AsynchronousSocketChannelImpl.write(AsynchronousSocketChannelImpl.java:382)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
