[jira] [Commented] (SSHD-85) Port Forward closes connection before all bytes are sent

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/SSHD-85?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16062330#comment-16062330
 ] 

ASF GitHub Bot commented on SSHD-85:


Github user bkuker closed the pull request at:

https://github.com/apache/mina-sshd/pull/33


> Port Forward closes connection before all bytes are sent
> 
>
> Key: SSHD-85
> URL: https://issues.apache.org/jira/browse/SSHD-85
> Project: MINA SSHD
>  Issue Type: Bug
>Affects Versions: 0.2.0, 1.4.0, 1.6.0
> Environment: Windows or Linux, JSCH Client
>Reporter: Bill Kuker
>Assignee: Goldstein Lyor
> Fix For: 1.6.0
>
> Attachments: MINA-Port-Forward.txt, SSHD-85_testcase.txt
>
>
> When I am forwarding connections from a Client to a port on the SSH Server 
> through the SSH Client to a Remote Server on the SSH Client network:
> Client --(a)---> SSH Server <===[ssh]===> SSH Client --(b)--> Remote Server
>  if the Remote Server sends some bytes and immediately closes the connection 
> (b) to the SSH Client the SSH Server will close the connection (a) to the 
> client before all of those bytes are sent to the SSH Client.
> This is probably a race condition. If I run one connection at a time it works 
> 99% of the time, but if I use a load generator (a completely separate 
> process) to max out all 4 cores I can make it fail (I do not get all the 
> bytes from Remote Server at Client) 99% of the time. Also by running about 
> 100 simultaneous requests I get maybe a 20% failure rate.
> If the Remote Server is something like Telnet (Open for a while, user closes 
> connection) everything seems fine, but the the Remote Server is a web server 
> with keep alive disabled, and many small requests are made, many images, 
> htmls, css are truncated.
> Sorry I can't be more precise with this, but I hope anyone else having a 
> similar trouble can help fill out this bug with details.
> *Update in 2017*
> I have provided a maven project with a unit test that has a 100% failure rate 
> on my desktop without resorting to creating artificial load. It is a lot 
> simpler:
> https://github.com/bkuker/sshd-85
> https://github.com/bkuker/sshd-85/blob/master/src/main/java/com/billkuker/sshd/sshd85/TheTest.java



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (SSHD-85) Port Forward closes connection before all bytes are sent

[Goldstein Lyor (JIRA)]

[ 
https://issues.apache.org/jira/browse/SSHD-85?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16059947#comment-16059947
 ] 

Goldstein Lyor commented on SSHD-85:


I have imported some of the code you posted and am getting unit-test failures. 
Please clone/fork https://github.com/lgoldstein/mina-sshd and select SSHD-85 
branch. You will find your original code with some minor changes that IMO 
should not have impacted them. 
{{org.apache.sshd.common.forward.AbstractServerCloseTestSupport.testLocalPortForwardLoop()}}
 fails when I run it in Eclipse - BTW, my environment is Windows 10 + Oracle 
Java 1.8.0_131
{noformat}
java.lang.AssertionError: Mismatched data length expected:<7800> but was:<0>
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:834)
at org.junit.Assert.assertEquals(Assert.java:645)
at 
org.apache.sshd.common.forward.AbstractServerCloseTestSupport.readInLoop(AbstractServerCloseTestSupport.java:127)
at 
org.apache.sshd.common.forward.AbstractServerCloseTestSupport.testLocalPortForwardLoop(AbstractServerCloseTestSupport.java:223)
{noformat}

> Port Forward closes connection before all bytes are sent
> 
>
> Key: SSHD-85
> URL: https://issues.apache.org/jira/browse/SSHD-85
> Project: MINA SSHD
>  Issue Type: Bug
>Affects Versions: 0.2.0, 1.4.0, 1.6.0
> Environment: Windows or Linux, JSCH Client
>Reporter: Bill Kuker
>Assignee: Goldstein Lyor
> Attachments: MINA-Port-Forward.txt, SSHD-85_testcase.txt
>
>
> When I am forwarding connections from a Client to a port on the SSH Server 
> through the SSH Client to a Remote Server on the SSH Client network:
> Client --(a)---> SSH Server <===[ssh]===> SSH Client --(b)--> Remote Server
>  if the Remote Server sends some bytes and immediately closes the connection 
> (b) to the SSH Client the SSH Server will close the connection (a) to the 
> client before all of those bytes are sent to the SSH Client.
> This is probably a race condition. If I run one connection at a time it works 
> 99% of the time, but if I use a load generator (a completely separate 
> process) to max out all 4 cores I can make it fail (I do not get all the 
> bytes from Remote Server at Client) 99% of the time. Also by running about 
> 100 simultaneous requests I get maybe a 20% failure rate.
> If the Remote Server is something like Telnet (Open for a while, user closes 
> connection) everything seems fine, but the the Remote Server is a web server 
> with keep alive disabled, and many small requests are made, many images, 
> htmls, css are truncated.
> Sorry I can't be more precise with this, but I hope anyone else having a 
> similar trouble can help fill out this bug with details.
> *Update in 2017*
> I have provided a maven project with a unit test that has a 100% failure rate 
> on my desktop without resorting to creating artificial load. It is a lot 
> simpler:
> https://github.com/bkuker/sshd-85
> https://github.com/bkuker/sshd-85/blob/master/src/main/java/com/billkuker/sshd/sshd85/TheTest.java



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (SSHD-85) Port Forward closes connection before all bytes are sent

[Goldstein Lyor (JIRA)]

[ 
https://issues.apache.org/jira/browse/SSHD-85?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16059667#comment-16059667
 ] 

Goldstein Lyor commented on SSHD-85:


Great - as soon as yo have made the recommended changes and {{mvn clean 
install}} passes please re-post and I will be more than happy to test and 
eventually commit it (with due credit) - we appreciate contributions such as 
yours.

> Port Forward closes connection before all bytes are sent
> 
>
> Key: SSHD-85
> URL: https://issues.apache.org/jira/browse/SSHD-85
> Project: MINA SSHD
>  Issue Type: Bug
>Affects Versions: 0.2.0, 1.4.0, 1.6.0
> Environment: Windows or Linux, JSCH Client
>Reporter: Bill Kuker
>Assignee: Guillaume Nodet
> Attachments: MINA-Port-Forward.txt, SSHD-85_testcase.txt
>
>
> When I am forwarding connections from a Client to a port on the SSH Server 
> through the SSH Client to a Remote Server on the SSH Client network:
> Client --(a)---> SSH Server <===[ssh]===> SSH Client --(b)--> Remote Server
>  if the Remote Server sends some bytes and immediately closes the connection 
> (b) to the SSH Client the SSH Server will close the connection (a) to the 
> client before all of those bytes are sent to the SSH Client.
> This is probably a race condition. If I run one connection at a time it works 
> 99% of the time, but if I use a load generator (a completely separate 
> process) to max out all 4 cores I can make it fail (I do not get all the 
> bytes from Remote Server at Client) 99% of the time. Also by running about 
> 100 simultaneous requests I get maybe a 20% failure rate.
> If the Remote Server is something like Telnet (Open for a while, user closes 
> connection) everything seems fine, but the the Remote Server is a web server 
> with keep alive disabled, and many small requests are made, many images, 
> htmls, css are truncated.
> Sorry I can't be more precise with this, but I hope anyone else having a 
> similar trouble can help fill out this bug with details.
> *Update in 2017*
> I have provided a maven project with a unit test that has a 100% failure rate 
> on my desktop without resorting to creating artificial load. It is a lot 
> simpler:
> https://github.com/bkuker/sshd-85
> https://github.com/bkuker/sshd-85/blob/master/src/main/java/com/billkuker/sshd/sshd85/TheTest.java



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (SSHD-85) Port Forward closes connection before all bytes are sent

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/SSHD-85?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16059561#comment-16059561
 ] 

ASF GitHub Bot commented on SSHD-85:


GitHub user bkuker opened a pull request:

https://github.com/apache/mina-sshd/pull/33

Tests & Candidate fix for SSHD-85.

SSHD-85 is a bug that occurs on Windows where the last few bytes of data 
sent through a port forward can be lost when the server at the far end of the 
port forward sends the data and immediately closes the socket.

An example of this server behavior would be HTTP 1.0 with the Connection: 
close header. The server sends the data, and then closes the socket to indicate 
that it is done sending data. The server does not send a Content-Length header 
or similar. This is a stupid way to do it, all modern http servers do better, 
but there are still old embedded systems that work this way, and it does impact 
my use case.

The test cases include some sleeps on the client end. This seems to be a 
race condition that occurs possibly one time in a hundred or more, but these 
sleeps cause it to happen every single time.
The test cases include two controls: JSCH as the client

During testing I did verify that the test server did send all data to the 
ssh client by examining the buffers in the ssh client and seeing all of the 
data. I then verified that all of the data was transferred from the ssh client 
to the ssh server by examining the buffers in the sshd server.  The sshd server 
does write all of that data to the socket as far as I can tell, but on Windows 
some still goes missing at the end! I also verified that on Local port forwards 
the error does occur with Apache SSHD as the client, and does NOT occur with 
JSCH as the client. I would like to try that the other way around with Apache 
SSH Client and another server, but the pickings for Java SSH servers are slim, 
and this one is the best!

I am currently trying to understand WHY the fix (basically one line, 
getSocket().shutdownOutput()) fixes the problem. I am submitting this PR now in 
hopes that someone with a better understanding will point out why it is the 
correct fix and accept it. All I can say at the moment is that it works.

Also includes a fix for a test in PortForwardingLoadTest which was failing 
for me before I started investigating. Frankly it looks like it never worked; I 
would guess that there was some copy-paste mistake in the past when it got 
updated to use try-with-resources.

I am ... very motivated to get this fixed so I do not have to maintain my 
own private branch of sshd, so please ask questions, make suggestions, if you 
think I am totally crazy help me prove that I am not!

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/bkuker/mina-sshd sshd-85

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/mina-sshd/pull/33.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #33


commit e6e400b24fe5d4d6778135e1ae5e38fb8ac3631a
Author: bkuker 
Date:   2017-06-14T12:20:41Z

Tests for SSHD-85

commit fe92dc673678e66374d2a095a2299b8e60425c98
Author: bkuker 
Date:   2017-06-20T15:01:56Z

The Fix?

commit 5842c850485835a98385937df7bb74b0ca4c4889
Author: bkuker 
Date:   2017-06-22T13:13:57Z

This test failed before my changes. It made no sense as it was.




> Port Forward closes connection before all bytes are sent
> 
>
> Key: SSHD-85
> URL: https://issues.apache.org/jira/browse/SSHD-85
> Project: MINA SSHD
>  Issue Type: Bug
>Affects Versions: 0.2.0, 1.4.0, 1.6.0
> Environment: Windows or Linux, JSCH Client
>Reporter: Bill Kuker
>Assignee: Guillaume Nodet
> Attachments: MINA-Port-Forward.txt, SSHD-85_testcase.txt
>
>
> When I am forwarding connections from a Client to a port on the SSH Server 
> through the SSH Client to a Remote Server on the SSH Client network:
> Client --(a)---> SSH Server <===[ssh]===> SSH Client --(b)--> Remote Server
>  if the Remote Server sends some bytes and immediately closes the connection 
> (b) to the SSH Client the SSH Server will close the connection (a) to the 
> client before all of those bytes are sent to the SSH Client.
> This is probably a race condition. If I run one connection at a time it works 
> 99% of the time, but if I use a load generator (a completely separate 
> process) to max out all 4 cores I can make it fail (I do not get all the 
> bytes from Remote Server at Client) 99% of the time. Also by running about 
> 100 simultaneous requests I 

[jira] [Commented] (SSHD-85) Port Forward closes connection before all bytes are sent

[Bill Kuker (JIRA)]

[ 
https://issues.apache.org/jira/browse/SSHD-85?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16057359#comment-16057359
 ] 

Bill Kuker commented on SSHD-85:


I solemnly swear I am going to kill this bug...

Narrowing down the problem, if I switch to JSCH as the client the Remote port 
forward tests continue to fail, but the local port forward all begin to pass.

So the bug seems to be present in apache ssh client and server, at the point 
where the apache ssh client or ssh server is talking to the test client, not 
the test server. I would guess that if I were to implement another version of 
the test with the apache ssh client and another server (I'll look to see if 
JSCH has one) the remote tests would fail.



> Port Forward closes connection before all bytes are sent
> 
>
> Key: SSHD-85
> URL: https://issues.apache.org/jira/browse/SSHD-85
> Project: MINA SSHD
>  Issue Type: Bug
>Affects Versions: 0.2.0, 1.4.0, 1.6.0
> Environment: Windows or Linux, JSCH Client
>Reporter: Bill Kuker
>Assignee: Guillaume Nodet
> Attachments: MINA-Port-Forward.txt, SSHD-85_testcase.txt
>
>
> When I am forwarding connections from a Client to a port on the SSH Server 
> through the SSH Client to a Remote Server on the SSH Client network:
> Client --(a)---> SSH Server <===[ssh]===> SSH Client --(b)--> Remote Server
>  if the Remote Server sends some bytes and immediately closes the connection 
> (b) to the SSH Client the SSH Server will close the connection (a) to the 
> client before all of those bytes are sent to the SSH Client.
> This is probably a race condition. If I run one connection at a time it works 
> 99% of the time, but if I use a load generator (a completely separate 
> process) to max out all 4 cores I can make it fail (I do not get all the 
> bytes from Remote Server at Client) 99% of the time. Also by running about 
> 100 simultaneous requests I get maybe a 20% failure rate.
> If the Remote Server is something like Telnet (Open for a while, user closes 
> connection) everything seems fine, but the the Remote Server is a web server 
> with keep alive disabled, and many small requests are made, many images, 
> htmls, css are truncated.
> Sorry I can't be more precise with this, but I hope anyone else having a 
> similar trouble can help fill out this bug with details.
> *Update in 2017*
> I have provided a maven project with a unit test that has a 100% failure rate 
> on my desktop without resorting to creating artificial load. It is a lot 
> simpler:
> https://github.com/bkuker/sshd-85
> https://github.com/bkuker/sshd-85/blob/master/src/main/java/com/billkuker/sshd/sshd85/TheTest.java



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (SSHD-85) Port Forward closes connection before all bytes are sent

[Goldstein Lyor (JIRA)]

[ 
https://issues.apache.org/jira/browse/SSHD-85?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16056214#comment-16056214
 ] 

Goldstein Lyor commented on SSHD-85:


Looks like you may be on the right track - keep up the good work.

> Port Forward closes connection before all bytes are sent
> 
>
> Key: SSHD-85
> URL: https://issues.apache.org/jira/browse/SSHD-85
> Project: MINA SSHD
>  Issue Type: Bug
>Affects Versions: 0.2.0, 1.4.0, 1.6.0
> Environment: Windows or Linux, JSCH Client
>Reporter: Bill Kuker
>Assignee: Guillaume Nodet
> Attachments: MINA-Port-Forward.txt, SSHD-85_testcase.txt
>
>
> When I am forwarding connections from a Client to a port on the SSH Server 
> through the SSH Client to a Remote Server on the SSH Client network:
> Client --(a)---> SSH Server <===[ssh]===> SSH Client --(b)--> Remote Server
>  if the Remote Server sends some bytes and immediately closes the connection 
> (b) to the SSH Client the SSH Server will close the connection (a) to the 
> client before all of those bytes are sent to the SSH Client.
> This is probably a race condition. If I run one connection at a time it works 
> 99% of the time, but if I use a load generator (a completely separate 
> process) to max out all 4 cores I can make it fail (I do not get all the 
> bytes from Remote Server at Client) 99% of the time. Also by running about 
> 100 simultaneous requests I get maybe a 20% failure rate.
> If the Remote Server is something like Telnet (Open for a while, user closes 
> connection) everything seems fine, but the the Remote Server is a web server 
> with keep alive disabled, and many small requests are made, many images, 
> htmls, css are truncated.
> Sorry I can't be more precise with this, but I hope anyone else having a 
> similar trouble can help fill out this bug with details.
> *Update in 2017*
> I have provided a maven project with a unit test that has a 100% failure rate 
> on my desktop without resorting to creating artificial load. It is a lot 
> simpler:
> https://github.com/bkuker/sshd-85
> https://github.com/bkuker/sshd-85/blob/master/src/main/java/com/billkuker/sshd/sshd85/TheTest.java



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (SSHD-85) Port Forward closes connection before all bytes are sent

[Bill Kuker (JIRA)]

[ 
https://issues.apache.org/jira/browse/SSHD-85?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16055905#comment-16055905
 ] 

Bill Kuker commented on SSHD-85:


I have a "NOT A FIX" fix

The following change to the server makes every single test I run pass every 
single time: 
https://github.com/bkuker/mina-sshd/commit/2e1ae7bdd93420039d0b06317b3377a03707a839

*My opinion on what that means:*

There is a race on the SSHD server between the final writes to the client and 
the connection closing. Delaying the close by an arbitrary amount of time means 
the write() call always wins the race.

*Why this "fix" is no good:*

Obviously a 500ms sleep is a silly way to fix the problem, and is still a race 
condition, just one with a nearly certain outcome.

*What I need to do:*

Delay the close() call until the final write() call is done. At that point all 
of the data have been passed on to the OS and TCP will do it's best to finish 
sending the data.

> Port Forward closes connection before all bytes are sent
> 
>
> Key: SSHD-85
> URL: https://issues.apache.org/jira/browse/SSHD-85
> Project: MINA SSHD
>  Issue Type: Bug
>Affects Versions: 0.2.0, 1.4.0, 1.6.0
> Environment: Windows or Linux, JSCH Client
>Reporter: Bill Kuker
>Assignee: Guillaume Nodet
> Attachments: MINA-Port-Forward.txt, SSHD-85_testcase.txt
>
>
> When I am forwarding connections from a Client to a port on the SSH Server 
> through the SSH Client to a Remote Server on the SSH Client network:
> Client --(a)---> SSH Server <===[ssh]===> SSH Client --(b)--> Remote Server
>  if the Remote Server sends some bytes and immediately closes the connection 
> (b) to the SSH Client the SSH Server will close the connection (a) to the 
> client before all of those bytes are sent to the SSH Client.
> This is probably a race condition. If I run one connection at a time it works 
> 99% of the time, but if I use a load generator (a completely separate 
> process) to max out all 4 cores I can make it fail (I do not get all the 
> bytes from Remote Server at Client) 99% of the time. Also by running about 
> 100 simultaneous requests I get maybe a 20% failure rate.
> If the Remote Server is something like Telnet (Open for a while, user closes 
> connection) everything seems fine, but the the Remote Server is a web server 
> with keep alive disabled, and many small requests are made, many images, 
> htmls, css are truncated.
> Sorry I can't be more precise with this, but I hope anyone else having a 
> similar trouble can help fill out this bug with details.
> *Update in 2017*
> I have provided a maven project with a unit test that has a 100% failure rate 
> on my desktop without resorting to creating artificial load. It is a lot 
> simpler:
> https://github.com/bkuker/sshd-85
> https://github.com/bkuker/sshd-85/blob/master/src/main/java/com/billkuker/sshd/sshd85/TheTest.java



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (SSHD-85) Port Forward closes connection before all bytes are sent

[Bill Kuker (JIRA)]

[ 
https://issues.apache.org/jira/browse/SSHD-85?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16052960#comment-16052960
 ] 

Bill Kuker commented on SSHD-85:


If the socket is not writable sshd waits to write the data, like it should, but 
the "close" code "jumps in line" because even if a socket can't be written to 
yet it can still be closed.

> Port Forward closes connection before all bytes are sent
> 
>
> Key: SSHD-85
> URL: https://issues.apache.org/jira/browse/SSHD-85
> Project: MINA SSHD
>  Issue Type: Bug
>Affects Versions: 0.2.0, 1.4.0, 1.6.0
> Environment: Windows or Linux, JSCH Client
>Reporter: Bill Kuker
>Assignee: Guillaume Nodet
> Attachments: MINA-Port-Forward.txt, SSHD-85_testcase.txt
>
>
> When I am forwarding connections from a Client to a port on the SSH Server 
> through the SSH Client to a Remote Server on the SSH Client network:
> Client --(a)---> SSH Server <===[ssh]===> SSH Client --(b)--> Remote Server
>  if the Remote Server sends some bytes and immediately closes the connection 
> (b) to the SSH Client the SSH Server will close the connection (a) to the 
> client before all of those bytes are sent to the SSH Client.
> This is probably a race condition. If I run one connection at a time it works 
> 99% of the time, but if I use a load generator (a completely separate 
> process) to max out all 4 cores I can make it fail (I do not get all the 
> bytes from Remote Server at Client) 99% of the time. Also by running about 
> 100 simultaneous requests I get maybe a 20% failure rate.
> If the Remote Server is something like Telnet (Open for a while, user closes 
> connection) everything seems fine, but the the Remote Server is a web server 
> with keep alive disabled, and many small requests are made, many images, 
> htmls, css are truncated.
> Sorry I can't be more precise with this, but I hope anyone else having a 
> similar trouble can help fill out this bug with details.
> *Update in 2017*
> I have provided a maven project with a unit test that has a 100% failure rate 
> on my desktop without resorting to creating artificial load. It is a lot 
> simpler:
> https://github.com/bkuker/sshd-85
> https://github.com/bkuker/sshd-85/blob/master/src/main/java/com/billkuker/sshd/sshd85/TheTest.java



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (SSHD-85) Port Forward closes connection before all bytes are sent

[Bill Kuker (JIRA)]

[ 
https://issues.apache.org/jira/browse/SSHD-85?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16052959#comment-16052959
 ] 

Bill Kuker commented on SSHD-85:


The behavior of the server is simplified, but more or less identical to, an 
http 1.0 server with connection: close in the header. It writes the bytes and 
then closes the connection.

Also my test fails all the time in Windows, but I have NOT gotten it to fail 
under linux. Can I ask what OS you have been running under? I am still working 
to try to reproduce under Linux. Experience tells me that code that is very 
broken on one OS is sometimes only very subtly broken on another; In 2010 when 
I first reported this bug less than one HTTP 1.0 request in a hundred failed 
when made via apache sshd...

The problem is that the server writes bytes and then closes, sshd gets those 
bytes and then sends them to the client. If the socket sshd is writing to is 
not ready to be written to when it comes time to pass along those bytes they 
are buffered in sshd. When it becomes ready to write sshd writes them... All 
good... BUT if the server closes the connection before sshd has written the 
bytes it has buffered [because the socket is not ready to be written to] sshd 
turns around and closes the socket to the client, having never written those 
bytes.

So the last part of the data is still in a buffer in sshd user space, read from 
the server, not written to the port forward client, when the server closes the 
connection... Sshd then closes the other socket, when what it needs to do is 
wait for the socket to be writable, write the bytes from it's buffer, then 
close it... Clear as mud?

> Port Forward closes connection before all bytes are sent
> 
>
> Key: SSHD-85
> URL: https://issues.apache.org/jira/browse/SSHD-85
> Project: MINA SSHD
>  Issue Type: Bug
>Affects Versions: 0.2.0, 1.4.0, 1.6.0
> Environment: Windows or Linux, JSCH Client
>Reporter: Bill Kuker
>Assignee: Guillaume Nodet
> Attachments: MINA-Port-Forward.txt, SSHD-85_testcase.txt
>
>
> When I am forwarding connections from a Client to a port on the SSH Server 
> through the SSH Client to a Remote Server on the SSH Client network:
> Client --(a)---> SSH Server <===[ssh]===> SSH Client --(b)--> Remote Server
>  if the Remote Server sends some bytes and immediately closes the connection 
> (b) to the SSH Client the SSH Server will close the connection (a) to the 
> client before all of those bytes are sent to the SSH Client.
> This is probably a race condition. If I run one connection at a time it works 
> 99% of the time, but if I use a load generator (a completely separate 
> process) to max out all 4 cores I can make it fail (I do not get all the 
> bytes from Remote Server at Client) 99% of the time. Also by running about 
> 100 simultaneous requests I get maybe a 20% failure rate.
> If the Remote Server is something like Telnet (Open for a while, user closes 
> connection) everything seems fine, but the the Remote Server is a web server 
> with keep alive disabled, and many small requests are made, many images, 
> htmls, css are truncated.
> Sorry I can't be more precise with this, but I hope anyone else having a 
> similar trouble can help fill out this bug with details.
> *Update in 2017*
> I have provided a maven project with a unit test that has a 100% failure rate 
> on my desktop without resorting to creating artificial load. It is a lot 
> simpler:
> https://github.com/bkuker/sshd-85
> https://github.com/bkuker/sshd-85/blob/master/src/main/java/com/billkuker/sshd/sshd85/TheTest.java



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (SSHD-85) Port Forward closes connection before all bytes are sent

[Goldstein Lyor (JIRA)]

[ 
https://issues.apache.org/jira/browse/SSHD-85?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16052705#comment-16052705
 ] 

Goldstein Lyor commented on SSHD-85:


I have been looking over the test code and have a few concerns - the main one 
is that the server that sends the payload does not wait to make sure that the 
bytes have been sent and successfully acknowledged by the receiver - it simply 
writes the payload and then closes the socket, AFAIK this does not guarantee 
that all the written bytes have been sent and successfully acknowledged by the 
receiver. Assuming this is the case, no wonder bytes are getting lost since it 
becomes a *race* condition. This can also explain why using a tunnel, a large 
buffer or having a long enough pause in read fails - the receiver/reader side 
eventually attempts to read from a closed socket. In order to avod that, I 
would recommend using an {{AsynchronousSocketChannel}} at the server code that 
writes the payload and wait for the write completion - e.g. [this example 
code|https://gist.github.com/ochinchina/72cc23220dc8a933fc46].

> Port Forward closes connection before all bytes are sent
> 
>
> Key: SSHD-85
> URL: https://issues.apache.org/jira/browse/SSHD-85
> Project: MINA SSHD
>  Issue Type: Bug
>Affects Versions: 0.2.0, 1.4.0, 1.6.0
> Environment: Windows or Linux, JSCH Client
>Reporter: Bill Kuker
>Assignee: Guillaume Nodet
> Attachments: MINA-Port-Forward.txt, SSHD-85_testcase.txt
>
>
> When I am forwarding connections from a Client to a port on the SSH Server 
> through the SSH Client to a Remote Server on the SSH Client network:
> Client --(a)---> SSH Server <===[ssh]===> SSH Client --(b)--> Remote Server
>  if the Remote Server sends some bytes and immediately closes the connection 
> (b) to the SSH Client the SSH Server will close the connection (a) to the 
> client before all of those bytes are sent to the SSH Client.
> This is probably a race condition. If I run one connection at a time it works 
> 99% of the time, but if I use a load generator (a completely separate 
> process) to max out all 4 cores I can make it fail (I do not get all the 
> bytes from Remote Server at Client) 99% of the time. Also by running about 
> 100 simultaneous requests I get maybe a 20% failure rate.
> If the Remote Server is something like Telnet (Open for a while, user closes 
> connection) everything seems fine, but the the Remote Server is a web server 
> with keep alive disabled, and many small requests are made, many images, 
> htmls, css are truncated.
> Sorry I can't be more precise with this, but I hope anyone else having a 
> similar trouble can help fill out this bug with details.
> *Update in 2017*
> I have provided a maven project with a unit test that has a 100% failure rate 
> on my desktop without resorting to creating artificial load. It is a lot 
> simpler:
> https://github.com/bkuker/sshd-85
> https://github.com/bkuker/sshd-85/blob/master/src/main/java/com/billkuker/sshd/sshd85/TheTest.java



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (SSHD-85) Port Forward closes connection before all bytes are sent

[Goldstein Lyor (JIRA)]

[ 
https://issues.apache.org/jira/browse/SSHD-85?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16052024#comment-16052024
 ] 

Goldstein Lyor commented on SSHD-85:


Great work - we have not been able to reproduce - even your own test 
{quote}Nearly always fails{quote}. Once you have a fix ready, please post a 
pull-request and remember to also include your tests...

> Port Forward closes connection before all bytes are sent
> 
>
> Key: SSHD-85
> URL: https://issues.apache.org/jira/browse/SSHD-85
> Project: MINA SSHD
>  Issue Type: Bug
>Affects Versions: 0.2.0, 1.4.0, 1.6.0
> Environment: Windows or Linux, JSCH Client
>Reporter: Bill Kuker
>Assignee: Guillaume Nodet
> Attachments: MINA-Port-Forward.txt, SSHD-85_testcase.txt
>
>
> When I am forwarding connections from a Client to a port on the SSH Server 
> through the SSH Client to a Remote Server on the SSH Client network:
> Client --(a)---> SSH Server <===[ssh]===> SSH Client --(b)--> Remote Server
>  if the Remote Server sends some bytes and immediately closes the connection 
> (b) to the SSH Client the SSH Server will close the connection (a) to the 
> client before all of those bytes are sent to the SSH Client.
> This is probably a race condition. If I run one connection at a time it works 
> 99% of the time, but if I use a load generator (a completely separate 
> process) to max out all 4 cores I can make it fail (I do not get all the 
> bytes from Remote Server at Client) 99% of the time. Also by running about 
> 100 simultaneous requests I get maybe a 20% failure rate.
> If the Remote Server is something like Telnet (Open for a while, user closes 
> connection) everything seems fine, but the the Remote Server is a web server 
> with keep alive disabled, and many small requests are made, many images, 
> htmls, css are truncated.
> Sorry I can't be more precise with this, but I hope anyone else having a 
> similar trouble can help fill out this bug with details.
> *Update in 2017*
> I have provided a maven project with a unit test that has a 100% failure rate 
> on my desktop without resorting to creating artificial load. It is a lot 
> simpler:
> https://github.com/bkuker/sshd-85
> https://github.com/bkuker/sshd-85/blob/master/src/main/java/com/billkuker/sshd/sshd85/TheTest.java



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (SSHD-85) Port Forward closes connection before all bytes are sent

[Bill Kuker (JIRA)]

[ 
https://issues.apache.org/jira/browse/SSHD-85?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16048266#comment-16048266
 ] 

Bill Kuker commented on SSHD-85:


Here is a more minimal reproduction...

https://github.com/bkuker/sshd-85

It creates an SSH Server, an SSH Client, and a Test Server.

The Test server writes a known string to the client and closes the connection.

There are 3 tests:

readFastPF: Connects to the test server via a port forward and reads everything 
at once. Always works.
readSlowDirect: Connects to the test server directly (NO port forward) and 
reads the data in two smaller buffers with a 50ms delay in between. Always 
works.
readSlowPF: Connects to the test server via a port forward and reads the data 
in two smaller buffers with a 50ms delay in between. (Nearly) always fails.

The problem is that the when the test server closes the socket Apache SSH 
closes the port forward *despite there being data left to send to the client*.

> Port Forward closes connection before all bytes are sent
> 
>
> Key: SSHD-85
> URL: https://issues.apache.org/jira/browse/SSHD-85
> Project: MINA SSHD
>  Issue Type: Bug
>Affects Versions: 0.2.0
> Environment: Windows or Linux, JSCH Client
>Reporter: Bill Kuker
>Assignee: Guillaume Nodet
> Attachments: MINA-Port-Forward.txt, SSHD-85_testcase.txt
>
>
> When I am forwarding connections from a Client to a port on the SSH Server 
> through the SSH Client to a Remote Server on the SSH Client network:
> Client --(a)---> SSH Server <===[ssh]===> SSH Client --(b)--> Remote Server
>  if the Remote Server sends some bytes and immediately closes the connection 
> (b) to the SSH Client the SSH Server will close the connection (a) to the 
> client before all of those bytes are sent to the SSH Client.
> This is probably a race condition. If I run one connection at a time it works 
> 99% of the time, but if I use a load generator (a completely separate 
> process) to max out all 4 cores I can make it fail (I do not get all the 
> bytes from Remote Server at Client) 99% of the time. Also by running about 
> 100 simultaneous requests I get maybe a 20% failure rate.
> If the Remote Server is something like Telnet (Open for a while, user closes 
> connection) everything seems fine, but the the Remote Server is a web server 
> with keep alive disabled, and many small requests are made, many images, 
> htmls, css are truncated.
> Sorry I can't be more precise with this, but I hope anyone else having a 
> similar trouble can help fill out this bug with details.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] Commented: (SSHD-85) Port Forward closes connection before all bytes are sent

[Guillaume Nodet (JIRA)]

[ 
https://issues.apache.org/jira/browse/SSHD-85?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12936094#action_12936094
 ] 

Guillaume Nodet commented on SSHD-85:
-

Where are we at with this issue ? Bill, did you ever raise a new JIRA to ask 
for a close() method does wait until pending writes are done ? Or is that maybe 
already available ?

 Port Forward closes connection before all bytes are sent
 

 Key: SSHD-85
 URL: https://issues.apache.org/jira/browse/SSHD-85
 Project: MINA SSHD
  Issue Type: Bug
Affects Versions: 0.2.0
 Environment: Windows or Linux, JSCH Client
Reporter: Bill Kuker
 Attachments: MINA-Port-Forward.txt


 When I am forwarding connections from a Client to a port on the SSH Server 
 through the SSH Client to a Remote Server on the SSH Client network:
 Client --(a)--- SSH Server ===[ssh]=== SSH Client --(b)-- Remote Server
  if the Remote Server sends some bytes and immediately closes the connection 
 (b) to the SSH Client the SSH Server will close the connection (a) to the 
 client before all of those bytes are sent to the SSH Client.
 This is probably a race condition. If I run one connection at a time it works 
 99% of the time, but if I use a load generator (a completely separate 
 process) to max out all 4 cores I can make it fail (I do not get all the 
 bytes from Remote Server at Client) 99% of the time. Also by running about 
 100 simultaneous requests I get maybe a 20% failure rate.
 If the Remote Server is something like Telnet (Open for a while, user closes 
 connection) everything seems fine, but the the Remote Server is a web server 
 with keep alive disabled, and many small requests are made, many images, 
 htmls, css are truncated.
 Sorry I can't be more precise with this, but I hope anyone else having a 
 similar trouble can help fill out this bug with details.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.