[
https://issues.apache.org/jira/browse/SSHD-1070?focusedWorklogId=479786&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-479786
]
ASF GitHub Bot logged work on SSHD-1070:
----------------------------------------
Author: ASF GitHub Bot
Created on: 07/Sep/20 21:03
Start Date: 07/Sep/20 21:03
Worklog Time Spent: 10m
Work Description: tomaswolf commented on a change in pull request #163:
URL: https://github.com/apache/mina-sshd/pull/163#discussion_r484566434
##########
File path:
sshd-core/src/main/java/org/apache/sshd/common/channel/BufferedIoOutputStream.java
##########
@@ -56,7 +56,12 @@ public IoWriteFuture writePacket(Buffer buffer) throws
IOException {
}
IoWriteFutureImpl future = new IoWriteFutureImpl(getId(), buffer);
- writes.add(future);
+ try {
+ writes.put(future);
+ } catch (InterruptedException e) {
+ Thread.interrupted();
+ throw new IOException(e);
+ }
Review comment:
I know about the semantics of `put`. Yes, it'll block indefinitely until
there is space again - which will be when the slow consumer has caught up. It
doesn't make any sense to let the producer continue any earlier. But maybe
there's a better place to implement this throttle than in this
`BufferedIoOutputStream`.
Your suggestion is a larger refactoring that I cannot do. Using `offer` with
timeout one also would have to handle in the caller the case that the internal
queue was full, and wait there, and there'd need to be a way to configure the
queue size. Moreover the queue size and timeout needed would depend on the
relative speeds of producer and consumer, which are unknown in general. I
wouldn't know how to configure such a thing such that it worked for all cases.
Just failing the transfer when the queue is ever full is IMO not an option; the
whole mechanism should probably still work (albeit slowly) with a very fast
producer and a very slow consumer.
The problem is precisely that current code doesn't place any restrictions,
and thus a fast producer can overrun a slow consumer so much that the server
gets an OOME.
But in any case SSHD-1070 and SSHD-1069 should probably be resolved together.
BTW, `Thread.interrupted()`is wrong; should have been
`Thread.currentThread().interrupt()`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 479786)
Time Spent: 50m (was: 40m)
> OutOfMemoryError when use port forwarding
> -----------------------------------------
>
> Key: SSHD-1070
> URL: https://issues.apache.org/jira/browse/SSHD-1070
> Project: MINA SSHD
> Issue Type: Bug
> Affects Versions: 2.5.1
> Reporter: Feng Jiajie
> Priority: Major
> Time Spent: 50m
> Remaining Estimate: 0h
>
> Hi [~gnodet], I found this commit implemented asynchronous port forwarding:
> [https://github.com/apache/mina-sshd/commit/45f84aab59b2e11d72942cffe9d810e37ab64959#diff-33823b8546f71d77bb1d653358ecde70]
> However, when a large amount of data is returned from upstream application,
> it is possible to cause an OOM in SSHD.
> Step1. SSHD server:
> {code:java}
> import org.apache.sshd.common.FactoryManager;
> import org.apache.sshd.common.PropertyResolverUtils;
> import org.apache.sshd.common.util.security.SecurityUtils;
> import org.apache.sshd.server.SshServer;
> import org.apache.sshd.server.forward.AcceptAllForwardingFilter;
> import org.apache.sshd.server.keyprovider.SimpleGeneratorHostKeyProvider;
> import java.io.IOException;
> import java.nio.file.Paths;
> public class TestSshd2 {
> public static void main(String[] args) throws IOException,
> InterruptedException {
> SecurityUtils.setAPrioriDisabledProvider("BC", true);
> SshServer sshd = SshServer.setUpDefaultServer();
> sshd.setPort(12133);
> sshd.setKeyPairProvider(new
> SimpleGeneratorHostKeyProvider(Paths.get("/tmp/aa.key")));
> sshd.setPasswordAuthenticator((username, password, session) -> true);
> sshd.setForwardingFilter(AcceptAllForwardingFilter.INSTANCE);
> sshd.start();
> Thread.sleep(100000000);
> }
> }
> {code}
> Step2. start ssh client and iperf3 server:
> {code:java}
> ssh -o 'ExitOnForwardFailure yes' -p 12133 -f -x -N -T -L
> 0.0.0.0:15678:127.0.0.1:12345 [email protected]
> iperf3 -s -p 12345
> {code}
> Step3. run iperf3 client:
> {code:java}
> iperf3 -c 127.0.0.1 -i 1 -t 120 -p 15678 -P 8 --reverse
> {code}
> *-R, --reverse run in reverse mode (server sends, client receives)*
> SSHD will receive a lot of data but will not be able to forward it in time.
> log when OOM:
> {code:java}
> 17:52:25.195 [sshd-SshServer[5bcea91b](port=12133)-nio2-thread-1] WARN
> org.apache.sshd.common.io.nio2.Nio2Session -
> exceptionCaught(Nio2Session[local=/127.0.0.1:33524, remote=/127.0.0.1:12345])
> Exception handler threw OutOfMemoryError, closing the session: GC overhead
> limit exceeded17:52:25.195
> [sshd-SshServer[5bcea91b](port=12133)-nio2-thread-1] WARN
> org.apache.sshd.common.io.nio2.Nio2Session -
> exceptionCaught(Nio2Session[local=/127.0.0.1:33524, remote=/127.0.0.1:12345])
> Exception handler threw OutOfMemoryError, closing the session: GC overhead
> limit exceeded at
> org.apache.sshd.common.io.nio2.Nio2CompletionHandler.completed(Nio2CompletionHandler.java:37)
> at sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:126) at
> sun.nio.ch.Invoker.invokeDirect(Invoker.java:157) at
> sun.nio.ch.UnixAsynchronousSocketChannelImpl.implWrite(UnixAsynchronousSocketChannelImpl.java:736)
> at
> sun.nio.ch.AsynchronousSocketChannelImpl.write(AsynchronousSocketChannelImpl.java:382)
> at
> sun.nio.ch.AsynchronousSocketChannelImpl.write(AsynchronousSocketChannelImpl.java:399)
> at
> org.apache.sshd.common.io.nio2.Nio2Session.doWriteCycle(Nio2Session.java:420)
> at
> org.apache.sshd.common.io.nio2.Nio2Session.startWriting(Nio2Session.java:404)
> at
> org.apache.sshd.common.io.nio2.Nio2Session.finishWrite(Nio2Session.java:495)
> at
> org.apache.sshd.common.io.nio2.Nio2Session.handleCompletedWriteCycle(Nio2Session.java:465)
> at
> org.apache.sshd.common.io.nio2.Nio2Session$2.onCompleted(Nio2Session.java:429)
> at
> org.apache.sshd.common.io.nio2.Nio2Session$2.onCompleted(Nio2Session.java:426)17:52:25.639
> [sshd-SshServer[5bcea91b](port=12133)-nio2-thread-6] WARN
> org.apache.sshd.common.io.nio2.Nio2Session -
> exceptionCaught(Nio2Session[local=/127.0.0.1:33530, remote=/127.0.0.1:12345])
> Exception handler threw OutOfMemoryError, closing the session: GC overhead
> limit exceeded at
> org.apache.sshd.common.io.nio2.Nio2CompletionHandler.lambda$completed$0(Nio2CompletionHandler.java:38)
> at java.security.AccessController.doPrivileged(Native Method) at
> org.apache.sshd.common.io.nio2.Nio2CompletionHandler.completed(Nio2CompletionHandler.java:37)
> at sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:126) at
> sun.nio.ch.Invoker.invokeDirect(Invoker.java:157) at
> sun.nio.ch.UnixAsynchronousSocketChannelImpl.implWrite(UnixAsynchronousSocketChannelImpl.java:736)17:52:26.045
> [sshd-SshServer[5bcea91b](port=12133)-nio2-thread-2] DEBUG
> org.apache.sshd.common.io.nio2.Nio2Session -
> exceptionCaught(Nio2Session[local=/127.0.0.1:33522, remote=/127.0.0.1:12345])
> caught OutOfMemoryError[GC overhead limit exceeded] - calling
> handler17:52:26.045 [sshd-SshServer[5bcea91b](port=12133)-nio2-thread-2]
> DEBUG org.apache.sshd.server.forward.TcpipServerChannel -
> exceptionCaught(TcpipServerChannel[id=3,
> recipient=4]-ServerSessionImpl[test5@/127.0.0.1:53702]) signal close
> immediately=false due to OutOfMemoryError[GC overhead limit
> exceeded]17:52:26.045 [sshd-SshServer[5bcea91b](port=12133)-nio2-thread-2]
> DEBUG org.apache.sshd.server.forward.TcpipServerChannel -
> close(TcpipServerChannel[id=3,
> recipient=4]-ServerSessionImpl[test5@/127.0.0.1:53702])[Graceful] state
> already Graceful17:52:26.794
> [sshd-SshServer[5bcea91b](port=12133)-nio2-thread-3] WARN
> org.apache.sshd.common.io.nio2.Nio2Session -
> exceptionCaught(Nio2Session[local=/127.0.0.1:33528, remote=/127.0.0.1:12345])
> Exception handler threw OutOfMemoryError, closing the session: GC overhead
> limit exceeded at
> sun.nio.ch.AsynchronousSocketChannelImpl.write(AsynchronousSocketChannelImpl.java:382)
> at
> sun.nio.ch.AsynchronousSocketChannelImpl.write(AsynchronousSocketChannelImpl.java:399)
> at
> org.apache.sshd.common.io.nio2.Nio2Session.doWriteCycle(Nio2Session.java:420)
> at
> org.apache.sshd.common.io.nio2.Nio2Session.startWriting(Nio2Session.java:404)
> at
> org.apache.sshd.common.io.nio2.Nio2Session.finishWrite(Nio2Session.java:495)
> at
> org.apache.sshd.common.io.nio2.Nio2Session.handleCompletedWriteCycle(Nio2Session.java:465)
> at
> org.apache.sshd.common.io.nio2.Nio2Session$2.onCompleted(Nio2Session.java:429)
> at
> org.apache.sshd.common.io.nio2.Nio2Session$2.onCompleted(Nio2Session.java:426)
> at
> org.apache.sshd.common.io.nio2.Nio2CompletionHandler.lambda$completed$0(Nio2CompletionHandler.java:38)17:52:26.045
> [sshd-SshServer[5bcea91b](port=12133)-nio2-thread-2] DEBUG
> org.apache.sshd.common.io.nio2.Nio2Session -
> close(Nio2Session[local=/127.0.0.1:33522, remote=/127.0.0.1:12345]) Closing
> immediately at java.security.AccessController.doPrivileged(Native
> Method)17:52:26.869 [sshd-SshServer[5bcea91b](port=12133)-nio2-thread-2]
> DEBUG org.apache.sshd.common.io.nio2.Nio2Session -
> doCloseImmediately(Nio2Session[local=/127.0.0.1:33522,
> remote=/127.0.0.1:12345]) closing
> socket=sun.nio.ch.UnixAsynchronousSocketChannelImpl[connected
> local=/127.0.0.1:33522 remote=/127.0.0.1:12345] at
> org.apache.sshd.common.io.nio2.Nio2CompletionHandler.completed(Nio2CompletionHandler.java:37)
> at sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:126) at
> sun.nio.ch.Invoker.invokeDirect(Invoker.java:157)17:52:27.568
> [sshd-SshServer[5bcea91b](port=12133)-nio2-thread-6] DEBUG
> org.apache.sshd.common.io.nio2.Nio2Session -
> exceptionCaught(Nio2Session[local=/127.0.0.1:33530, remote=/127.0.0.1:12345])
> caught OutOfMemoryError[GC overhead limit exceeded] - calling
> handler17:52:27.568 [sshd-SshServer[5bcea91b](port=12133)-nio2-thread-6]
> DEBUG org.apache.sshd.server.forward.TcpipServerChannel -
> exceptionCaught(TcpipServerChannel[id=7,
> recipient=8]-ServerSessionImpl[test5@/127.0.0.1:53702]) signal close
> immediately=false due to OutOfMemoryError[GC overhead limit
> exceeded]17:52:27.568 [sshd-SshServer[5bcea91b](port=12133)-nio2-thread-6]
> DEBUG org.apache.sshd.server.forward.TcpipServerChannel -
> close(TcpipServerChannel[id=7,
> recipient=8]-ServerSessionImpl[test5@/127.0.0.1:53702])[Graceful] state
> already Graceful at
> sun.nio.ch.UnixAsynchronousSocketChannelImpl.implWrite(UnixAsynchronousSocketChannelImpl.java:736)
> at
> sun.nio.ch.AsynchronousSocketChannelImpl.write(AsynchronousSocketChannelImpl.java:382)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
