[jira] [Commented] (SSHD-725) SFTP - Some files truncated on retrieval/download
[
https://issues.apache.org/jira/browse/SSHD-725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16001655#comment-16001655
]
Sue Kerschbaumer commented on SSHD-725:
---
This appears to be working with 1.5.0-SNAPSHOT. Thank you!
> SFTP - Some files truncated on retrieval/download
> -
>
> Key: SSHD-725
> URL: https://issues.apache.org/jira/browse/SSHD-725
> Project: MINA SSHD
> Issue Type: Bug
>Affects Versions: 1.0.0, 1.2.0, 1.3.0
> Environment: Various flavors of Linux (ubuntu xenial, centos 6, AWS
> AMI 2016.x)
>Reporter: Russ Tennant
>Assignee: Goldstein Lyor
> Attachments: download-lftp-script-output-trace.txt, test-download.data
>
>
> When retrieving files (depending on file size), the files are truncated. We
> have also experienced a similar problem on file upload where we get extra
> data repeated from the last packet received.
> I don't have a test-case for the file upload issue - yet. Still trying to
> find the magic file size.
> I created a github project that demonstrates the truncation issue on
> retrieval.
> Project: https://github.com/VentureTech/sshd-test
> Example lftp Script:
> https://github.com/VentureTech/sshd-test/blob/master/download-lftp-script.txt
> Example lftp Script Output:
> https://github.com/VentureTech/sshd-test/blob/master/download-lftp-script-output.txt
> I used lftp to demonstrate the issue, but it happens with all clients I've
> tried (openssh, Intellij IDEA, kde's sftp ioslave).
> There is a gradle task lftpDownload that will start the server and run the
> lftp script.
> {code}./gradlew lftpDownload{code}
> Attached is the data file I used to trigger the issue.
> In the referenced example output lftp terminates the retrieval with the
> following
> {code}
> [116371] 2017-01-16 15:27:32 localhost <--- got a packet, length=120,
> type=101(STATUS), id=12
> [116371] 2017-01-16 15:27:32 localhost status code=1(EOF),
> message=Internal EOFException: Unable to read 16384 bytes from offset=65536
> of 5c18fc884da4646cb80a4b12519e6f16
> [116371] 2017-01-16 15:27:32 localhost eof
> [116371] 2017-01-16 15:27:32 ---> sending a packet, length=41, type=4(CLOSE),
> id=15
> [116371] 2017-01-16 15:27:32 expected pos=61564, actual pos=49152
> mget: test-download.data: file size decreased during transfer
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (SSHD-725) SFTP - Some files truncated on retrieval/download
[
https://issues.apache.org/jira/browse/SSHD-725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16001060#comment-16001060
]
Goldstein Lyor commented on SSHD-725:
-
{quote}
FYI unused import in one of the sshd::git classes
{quote}
Thanks, fixed
> SFTP - Some files truncated on retrieval/download
> -
>
> Key: SSHD-725
> URL: https://issues.apache.org/jira/browse/SSHD-725
> Project: MINA SSHD
> Issue Type: Bug
>Affects Versions: 1.0.0, 1.2.0, 1.3.0
> Environment: Various flavors of Linux (ubuntu xenial, centos 6, AWS
> AMI 2016.x)
>Reporter: Russ Tennant
>Assignee: Goldstein Lyor
> Attachments: download-lftp-script-output-trace.txt, test-download.data
>
>
> When retrieving files (depending on file size), the files are truncated. We
> have also experienced a similar problem on file upload where we get extra
> data repeated from the last packet received.
> I don't have a test-case for the file upload issue - yet. Still trying to
> find the magic file size.
> I created a github project that demonstrates the truncation issue on
> retrieval.
> Project: https://github.com/VentureTech/sshd-test
> Example lftp Script:
> https://github.com/VentureTech/sshd-test/blob/master/download-lftp-script.txt
> Example lftp Script Output:
> https://github.com/VentureTech/sshd-test/blob/master/download-lftp-script-output.txt
> I used lftp to demonstrate the issue, but it happens with all clients I've
> tried (openssh, Intellij IDEA, kde's sftp ioslave).
> There is a gradle task lftpDownload that will start the server and run the
> lftp script.
> {code}./gradlew lftpDownload{code}
> Attached is the data file I used to trigger the issue.
> In the referenced example output lftp terminates the retrieval with the
> following
> {code}
> [116371] 2017-01-16 15:27:32 localhost <--- got a packet, length=120,
> type=101(STATUS), id=12
> [116371] 2017-01-16 15:27:32 localhost status code=1(EOF),
> message=Internal EOFException: Unable to read 16384 bytes from offset=65536
> of 5c18fc884da4646cb80a4b12519e6f16
> [116371] 2017-01-16 15:27:32 localhost eof
> [116371] 2017-01-16 15:27:32 ---> sending a packet, length=41, type=4(CLOSE),
> id=15
> [116371] 2017-01-16 15:27:32 expected pos=61564, actual pos=49152
> mget: test-download.data: file size decreased during transfer
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (SSHD-725) SFTP - Some files truncated on retrieval/download
[
https://issues.apache.org/jira/browse/SSHD-725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1506#comment-1506
]
Goldstein Lyor commented on SSHD-725:
-
It may be a different way on how EOF is being detected. Many clients assume
that if they asked to read _N_ bytes but the response contains _K < N_ bytes
then EOF is assumed. This was encountered during work on SSHD-728. This was
fixed in 1.5.0 (still to be released), but the issue also contains a possible
*workaround*:
{quote}
there is a quick fix - configure the server's sftp-max-packet-length value to
be higher than the size requested by the client - e.g.:
// Add this line to the server initialization code
PropertyResolverUtils.updateProperty(sshd,
SftpSubsystem.MAX_PACKET_LENGTH_PROP, 36 * 1024);
{quote}
> SFTP - Some files truncated on retrieval/download
> -
>
> Key: SSHD-725
> URL: https://issues.apache.org/jira/browse/SSHD-725
> Project: MINA SSHD
> Issue Type: Bug
>Affects Versions: 1.0.0, 1.2.0, 1.3.0
> Environment: Various flavors of Linux (ubuntu xenial, centos 6, AWS
> AMI 2016.x)
>Reporter: Russ Tennant
>Assignee: Goldstein Lyor
> Attachments: download-lftp-script-output-trace.txt, test-download.data
>
>
> When retrieving files (depending on file size), the files are truncated. We
> have also experienced a similar problem on file upload where we get extra
> data repeated from the last packet received.
> I don't have a test-case for the file upload issue - yet. Still trying to
> find the magic file size.
> I created a github project that demonstrates the truncation issue on
> retrieval.
> Project: https://github.com/VentureTech/sshd-test
> Example lftp Script:
> https://github.com/VentureTech/sshd-test/blob/master/download-lftp-script.txt
> Example lftp Script Output:
> https://github.com/VentureTech/sshd-test/blob/master/download-lftp-script-output.txt
> I used lftp to demonstrate the issue, but it happens with all clients I've
> tried (openssh, Intellij IDEA, kde's sftp ioslave).
> There is a gradle task lftpDownload that will start the server and run the
> lftp script.
> {code}./gradlew lftpDownload{code}
> Attached is the data file I used to trigger the issue.
> In the referenced example output lftp terminates the retrieval with the
> following
> {code}
> [116371] 2017-01-16 15:27:32 localhost <--- got a packet, length=120,
> type=101(STATUS), id=12
> [116371] 2017-01-16 15:27:32 localhost status code=1(EOF),
> message=Internal EOFException: Unable to read 16384 bytes from offset=65536
> of 5c18fc884da4646cb80a4b12519e6f16
> [116371] 2017-01-16 15:27:32 localhost eof
> [116371] 2017-01-16 15:27:32 ---> sending a packet, length=41, type=4(CLOSE),
> id=15
> [116371] 2017-01-16 15:27:32 expected pos=61564, actual pos=49152
> mget: test-download.data: file size decreased during transfer
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (SSHD-725) SFTP - Some files truncated on retrieval/download
[
https://issues.apache.org/jira/browse/SSHD-725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15999625#comment-15999625
]
Sue Kerschbaumer commented on SSHD-725:
---
I am having the same problem with a Mina SFTP server built on 1.4.0. running on
linux. Same exact issue with same output in the logs. I noticed that by
contrast a standard linux sftp server with the same lftp client uses a
different protocol (3 vs. 4) and sends larger data packets (but the same number
of packets). Is there a default compression happening to these files on
transport?
> SFTP - Some files truncated on retrieval/download
> -
>
> Key: SSHD-725
> URL: https://issues.apache.org/jira/browse/SSHD-725
> Project: MINA SSHD
> Issue Type: Bug
>Affects Versions: 1.0.0, 1.2.0, 1.3.0
> Environment: Various flavors of Linux (ubuntu xenial, centos 6, AWS
> AMI 2016.x)
>Reporter: Russ Tennant
>Assignee: Goldstein Lyor
> Attachments: download-lftp-script-output-trace.txt, test-download.data
>
>
> When retrieving files (depending on file size), the files are truncated. We
> have also experienced a similar problem on file upload where we get extra
> data repeated from the last packet received.
> I don't have a test-case for the file upload issue - yet. Still trying to
> find the magic file size.
> I created a github project that demonstrates the truncation issue on
> retrieval.
> Project: https://github.com/VentureTech/sshd-test
> Example lftp Script:
> https://github.com/VentureTech/sshd-test/blob/master/download-lftp-script.txt
> Example lftp Script Output:
> https://github.com/VentureTech/sshd-test/blob/master/download-lftp-script-output.txt
> I used lftp to demonstrate the issue, but it happens with all clients I've
> tried (openssh, Intellij IDEA, kde's sftp ioslave).
> There is a gradle task lftpDownload that will start the server and run the
> lftp script.
> {code}./gradlew lftpDownload{code}
> Attached is the data file I used to trigger the issue.
> In the referenced example output lftp terminates the retrieval with the
> following
> {code}
> [116371] 2017-01-16 15:27:32 localhost <--- got a packet, length=120,
> type=101(STATUS), id=12
> [116371] 2017-01-16 15:27:32 localhost status code=1(EOF),
> message=Internal EOFException: Unable to read 16384 bytes from offset=65536
> of 5c18fc884da4646cb80a4b12519e6f16
> [116371] 2017-01-16 15:27:32 localhost eof
> [116371] 2017-01-16 15:27:32 ---> sending a packet, length=41, type=4(CLOSE),
> id=15
> [116371] 2017-01-16 15:27:32 expected pos=61564, actual pos=49152
> mget: test-download.data: file size decreased during transfer
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (SSHD-725) SFTP - Some files truncated on retrieval/download
[
https://issues.apache.org/jira/browse/SSHD-725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15962114#comment-15962114
]
Goldstein Lyor commented on SSHD-725:
-
Sorry, still cannot reproduce (at least with the latest version). I have looked
at the demo test code you posted and I have many reservations (which I have
already listed) - to which I can also add the:
{code:java}
FileSystem fileSystem = newFileSystem(testDir.toPath(),
SftpServer.class.getClassLoader());
FileSystemFactory fileSystemFactory = session -> fileSystem;
sshd.setFileSystemFactory(fileSystemFactory);
{code}
In order to determine whether this is indeed a bug, I am renewing my request to:
* Use the latest version (1.5.0-SNAPSHOT)
* Run the {{SshServerMain#main}} code and see if it sill truncates the files.
> SFTP - Some files truncated on retrieval/download
> -
>
> Key: SSHD-725
> URL: https://issues.apache.org/jira/browse/SSHD-725
> Project: MINA SSHD
> Issue Type: Bug
>Affects Versions: 1.0.0, 1.2.0, 1.3.0
> Environment: Various flavors of Linux (ubuntu xenial, centos 6, AWS
> AMI 2016.x)
>Reporter: Russ Tennant
>Assignee: Goldstein Lyor
> Attachments: download-lftp-script-output-trace.txt, test-download.data
>
>
> When retrieving files (depending on file size), the files are truncated. We
> have also experienced a similar problem on file upload where we get extra
> data repeated from the last packet received.
> I don't have a test-case for the file upload issue - yet. Still trying to
> find the magic file size.
> I created a github project that demonstrates the truncation issue on
> retrieval.
> Project: https://github.com/VentureTech/sshd-test
> Example lftp Script:
> https://github.com/VentureTech/sshd-test/blob/master/download-lftp-script.txt
> Example lftp Script Output:
> https://github.com/VentureTech/sshd-test/blob/master/download-lftp-script-output.txt
> I used lftp to demonstrate the issue, but it happens with all clients I've
> tried (openssh, Intellij IDEA, kde's sftp ioslave).
> There is a gradle task lftpDownload that will start the server and run the
> lftp script.
> {code}./gradlew lftpDownload{code}
> Attached is the data file I used to trigger the issue.
> In the referenced example output lftp terminates the retrieval with the
> following
> {code}
> [116371] 2017-01-16 15:27:32 localhost <--- got a packet, length=120,
> type=101(STATUS), id=12
> [116371] 2017-01-16 15:27:32 localhost status code=1(EOF),
> message=Internal EOFException: Unable to read 16384 bytes from offset=65536
> of 5c18fc884da4646cb80a4b12519e6f16
> [116371] 2017-01-16 15:27:32 localhost eof
> [116371] 2017-01-16 15:27:32 ---> sending a packet, length=41, type=4(CLOSE),
> id=15
> [116371] 2017-01-16 15:27:32 expected pos=61564, actual pos=49152
> mget: test-download.data: file size decreased during transfer
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (SSHD-725) SFTP - Some files truncated on retrieval/download
[
https://issues.apache.org/jira/browse/SSHD-725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15962108#comment-15962108
]
Goldstein Lyor commented on SSHD-725:
-
Thanks, I'll see if I can reproduce it somehow
> SFTP - Some files truncated on retrieval/download
> -
>
> Key: SSHD-725
> URL: https://issues.apache.org/jira/browse/SSHD-725
> Project: MINA SSHD
> Issue Type: Bug
>Affects Versions: 1.0.0, 1.2.0, 1.3.0
> Environment: Various flavors of Linux (ubuntu xenial, centos 6, AWS
> AMI 2016.x)
>Reporter: Russ Tennant
>Assignee: Goldstein Lyor
> Attachments: download-lftp-script-output-trace.txt, test-download.data
>
>
> When retrieving files (depending on file size), the files are truncated. We
> have also experienced a similar problem on file upload where we get extra
> data repeated from the last packet received.
> I don't have a test-case for the file upload issue - yet. Still trying to
> find the magic file size.
> I created a github project that demonstrates the truncation issue on
> retrieval.
> Project: https://github.com/VentureTech/sshd-test
> Example lftp Script:
> https://github.com/VentureTech/sshd-test/blob/master/download-lftp-script.txt
> Example lftp Script Output:
> https://github.com/VentureTech/sshd-test/blob/master/download-lftp-script-output.txt
> I used lftp to demonstrate the issue, but it happens with all clients I've
> tried (openssh, Intellij IDEA, kde's sftp ioslave).
> There is a gradle task lftpDownload that will start the server and run the
> lftp script.
> {code}./gradlew lftpDownload{code}
> Attached is the data file I used to trigger the issue.
> In the referenced example output lftp terminates the retrieval with the
> following
> {code}
> [116371] 2017-01-16 15:27:32 localhost <--- got a packet, length=120,
> type=101(STATUS), id=12
> [116371] 2017-01-16 15:27:32 localhost status code=1(EOF),
> message=Internal EOFException: Unable to read 16384 bytes from offset=65536
> of 5c18fc884da4646cb80a4b12519e6f16
> [116371] 2017-01-16 15:27:32 localhost eof
> [116371] 2017-01-16 15:27:32 ---> sending a packet, length=41, type=4(CLOSE),
> id=15
> [116371] 2017-01-16 15:27:32 expected pos=61564, actual pos=49152
> mget: test-download.data: file size decreased during transfer
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (SSHD-725) SFTP - Some files truncated on retrieval/download
[
https://issues.apache.org/jira/browse/SSHD-725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15955520#comment-15955520
]
Russ Tennant commented on SSHD-725:
---
I included one file in the test project:
https://github.com/VentureTech/sshd-test/blob/master/data/test-download.data
It is 61564 bytes in length
It's reproducible 100% of the time with that file.
{code:title=Java Version}
java version "1.8.0_111"
Java(TM) SE Runtime Environment (build 1.8.0_111-b14)
Java HotSpot(TM) 64-Bit Server VM (build 25.111-b14, mixed mode)
{code}
{code:title=O/S}
ec2 - https://aws.amazon.com/amazon-linux-ami/2016.09-release-notes/
and locally on
Distributor ID: Ubuntu
Description:Ubuntu 16.10
Release:16.10
Codename: yakkety
{code}
In case of AWS, it's an EBS volume.
In case of local machine, it's an SSD drive.
There's no symlinks or remote mounts like NFS.
> SFTP - Some files truncated on retrieval/download
> -
>
> Key: SSHD-725
> URL: https://issues.apache.org/jira/browse/SSHD-725
> Project: MINA SSHD
> Issue Type: Bug
>Affects Versions: 1.0.0, 1.2.0, 1.3.0
> Environment: Various flavors of Linux (ubuntu xenial, centos 6, AWS
> AMI 2016.x)
>Reporter: Russ Tennant
>Assignee: Goldstein Lyor
> Attachments: download-lftp-script-output-trace.txt, test-download.data
>
>
> When retrieving files (depending on file size), the files are truncated. We
> have also experienced a similar problem on file upload where we get extra
> data repeated from the last packet received.
> I don't have a test-case for the file upload issue - yet. Still trying to
> find the magic file size.
> I created a github project that demonstrates the truncation issue on
> retrieval.
> Project: https://github.com/VentureTech/sshd-test
> Example lftp Script:
> https://github.com/VentureTech/sshd-test/blob/master/download-lftp-script.txt
> Example lftp Script Output:
> https://github.com/VentureTech/sshd-test/blob/master/download-lftp-script-output.txt
> I used lftp to demonstrate the issue, but it happens with all clients I've
> tried (openssh, Intellij IDEA, kde's sftp ioslave).
> There is a gradle task lftpDownload that will start the server and run the
> lftp script.
> {code}./gradlew lftpDownload{code}
> Attached is the data file I used to trigger the issue.
> In the referenced example output lftp terminates the retrieval with the
> following
> {code}
> [116371] 2017-01-16 15:27:32 localhost <--- got a packet, length=120,
> type=101(STATUS), id=12
> [116371] 2017-01-16 15:27:32 localhost status code=1(EOF),
> message=Internal EOFException: Unable to read 16384 bytes from offset=65536
> of 5c18fc884da4646cb80a4b12519e6f16
> [116371] 2017-01-16 15:27:32 localhost eof
> [116371] 2017-01-16 15:27:32 ---> sending a packet, length=41, type=4(CLOSE),
> id=15
> [116371] 2017-01-16 15:27:32 expected pos=61564, actual pos=49152
> mget: test-download.data: file size decreased during transfer
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (SSHD-725) SFTP - Some files truncated on retrieval/download
[
https://issues.apache.org/jira/browse/SSHD-725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15955502#comment-15955502
]
Goldstein Lyor commented on SSHD-725:
-
I have tried the code you mention - though not exactly as it appears - and it
worked without a hitch.
{quote}
none of them work with files of a certain size
{quote}
- What size(s) (*exactly*) ?
- Is it reproducible each and every time ?
- What environment are you using (O/S, Java version) ?
- Is there anything special about the files - e.g., symlinks, NFS mount, other ?
> SFTP - Some files truncated on retrieval/download
> -
>
> Key: SSHD-725
> URL: https://issues.apache.org/jira/browse/SSHD-725
> Project: MINA SSHD
> Issue Type: Bug
>Affects Versions: 1.0.0, 1.2.0, 1.3.0
> Environment: Various flavors of Linux (ubuntu xenial, centos 6, AWS
> AMI 2016.x)
>Reporter: Russ Tennant
>Assignee: Goldstein Lyor
> Attachments: download-lftp-script-output-trace.txt, test-download.data
>
>
> When retrieving files (depending on file size), the files are truncated. We
> have also experienced a similar problem on file upload where we get extra
> data repeated from the last packet received.
> I don't have a test-case for the file upload issue - yet. Still trying to
> find the magic file size.
> I created a github project that demonstrates the truncation issue on
> retrieval.
> Project: https://github.com/VentureTech/sshd-test
> Example lftp Script:
> https://github.com/VentureTech/sshd-test/blob/master/download-lftp-script.txt
> Example lftp Script Output:
> https://github.com/VentureTech/sshd-test/blob/master/download-lftp-script-output.txt
> I used lftp to demonstrate the issue, but it happens with all clients I've
> tried (openssh, Intellij IDEA, kde's sftp ioslave).
> There is a gradle task lftpDownload that will start the server and run the
> lftp script.
> {code}./gradlew lftpDownload{code}
> Attached is the data file I used to trigger the issue.
> In the referenced example output lftp terminates the retrieval with the
> following
> {code}
> [116371] 2017-01-16 15:27:32 localhost <--- got a packet, length=120,
> type=101(STATUS), id=12
> [116371] 2017-01-16 15:27:32 localhost status code=1(EOF),
> message=Internal EOFException: Unable to read 16384 bytes from offset=65536
> of 5c18fc884da4646cb80a4b12519e6f16
> [116371] 2017-01-16 15:27:32 localhost eof
> [116371] 2017-01-16 15:27:32 ---> sending a packet, length=41, type=4(CLOSE),
> id=15
> [116371] 2017-01-16 15:27:32 expected pos=61564, actual pos=49152
> mget: test-download.data: file size decreased during transfer
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (SSHD-725) SFTP - Some files truncated on retrieval/download
[
https://issues.apache.org/jira/browse/SSHD-725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15955453#comment-15955453
]
Russ Tennant commented on SSHD-725:
---
I've tried with 1.4.0 as well. Same issue. Have you tried reproducing with the
code from https://github.com/VentureTech/sshd-test ?
> SFTP - Some files truncated on retrieval/download
> -
>
> Key: SSHD-725
> URL: https://issues.apache.org/jira/browse/SSHD-725
> Project: MINA SSHD
> Issue Type: Bug
>Affects Versions: 1.0.0, 1.2.0, 1.3.0
> Environment: Various flavors of Linux (ubuntu xenial, centos 6, AWS
> AMI 2016.x)
>Reporter: Russ Tennant
>Assignee: Goldstein Lyor
> Attachments: download-lftp-script-output-trace.txt, test-download.data
>
>
> When retrieving files (depending on file size), the files are truncated. We
> have also experienced a similar problem on file upload where we get extra
> data repeated from the last packet received.
> I don't have a test-case for the file upload issue - yet. Still trying to
> find the magic file size.
> I created a github project that demonstrates the truncation issue on
> retrieval.
> Project: https://github.com/VentureTech/sshd-test
> Example lftp Script:
> https://github.com/VentureTech/sshd-test/blob/master/download-lftp-script.txt
> Example lftp Script Output:
> https://github.com/VentureTech/sshd-test/blob/master/download-lftp-script-output.txt
> I used lftp to demonstrate the issue, but it happens with all clients I've
> tried (openssh, Intellij IDEA, kde's sftp ioslave).
> There is a gradle task lftpDownload that will start the server and run the
> lftp script.
> {code}./gradlew lftpDownload{code}
> Attached is the data file I used to trigger the issue.
> In the referenced example output lftp terminates the retrieval with the
> following
> {code}
> [116371] 2017-01-16 15:27:32 localhost <--- got a packet, length=120,
> type=101(STATUS), id=12
> [116371] 2017-01-16 15:27:32 localhost status code=1(EOF),
> message=Internal EOFException: Unable to read 16384 bytes from offset=65536
> of 5c18fc884da4646cb80a4b12519e6f16
> [116371] 2017-01-16 15:27:32 localhost eof
> [116371] 2017-01-16 15:27:32 ---> sending a packet, length=41, type=4(CLOSE),
> id=15
> [116371] 2017-01-16 15:27:32 expected pos=61564, actual pos=49152
> mget: test-download.data: file size decreased during transfer
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (SSHD-725) SFTP - Some files truncated on retrieval/download
[
https://issues.apache.org/jira/browse/SSHD-725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15955424#comment-15955424
]
Goldstein Lyor commented on SSHD-725:
-
I have not been able to reproduce - however, please try the official 1.4.0
release as well as the [latest snaphot|https://github.com/apache/mina-sshd/]
> SFTP - Some files truncated on retrieval/download
> -
>
> Key: SSHD-725
> URL: https://issues.apache.org/jira/browse/SSHD-725
> Project: MINA SSHD
> Issue Type: Bug
>Affects Versions: 1.0.0, 1.2.0, 1.3.0
> Environment: Various flavors of Linux (ubuntu xenial, centos 6, AWS
> AMI 2016.x)
>Reporter: Russ Tennant
>Assignee: Goldstein Lyor
> Attachments: download-lftp-script-output-trace.txt, test-download.data
>
>
> When retrieving files (depending on file size), the files are truncated. We
> have also experienced a similar problem on file upload where we get extra
> data repeated from the last packet received.
> I don't have a test-case for the file upload issue - yet. Still trying to
> find the magic file size.
> I created a github project that demonstrates the truncation issue on
> retrieval.
> Project: https://github.com/VentureTech/sshd-test
> Example lftp Script:
> https://github.com/VentureTech/sshd-test/blob/master/download-lftp-script.txt
> Example lftp Script Output:
> https://github.com/VentureTech/sshd-test/blob/master/download-lftp-script-output.txt
> I used lftp to demonstrate the issue, but it happens with all clients I've
> tried (openssh, Intellij IDEA, kde's sftp ioslave).
> There is a gradle task lftpDownload that will start the server and run the
> lftp script.
> {code}./gradlew lftpDownload{code}
> Attached is the data file I used to trigger the issue.
> In the referenced example output lftp terminates the retrieval with the
> following
> {code}
> [116371] 2017-01-16 15:27:32 localhost <--- got a packet, length=120,
> type=101(STATUS), id=12
> [116371] 2017-01-16 15:27:32 localhost status code=1(EOF),
> message=Internal EOFException: Unable to read 16384 bytes from offset=65536
> of 5c18fc884da4646cb80a4b12519e6f16
> [116371] 2017-01-16 15:27:32 localhost eof
> [116371] 2017-01-16 15:27:32 ---> sending a packet, length=41, type=4(CLOSE),
> id=15
> [116371] 2017-01-16 15:27:32 expected pos=61564, actual pos=49152
> mget: test-download.data: file size decreased during transfer
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (SSHD-725) SFTP - Some files truncated on retrieval/download
[
https://issues.apache.org/jira/browse/SSHD-725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15954148#comment-15954148
]
Russ Tennant commented on SSHD-725:
---
[~lgoldstein] I built https://github.com/lgoldstein/mina-sshd/tree/SSHD-725 and
ran the test again. It still fails with the same message:
{code}
15:50:21.942 DEBUG o.a.s.s.subsystem.sftp.SftpSubsystem -
doSendStatus(ServerSessionImpl[test@/127.0.0.1:33512])[id=13] SSH_FXP_STATUS
(substatus=SSH_FX_EOF, lang=, msg=Internal EOFException: Unable to read 16384
bytes from offset=98304 of 3d8dcf9f9130bb97317d20b84b01c72c)
[19988] 2017-04-03 15:50:21 expected pos=61564, actual pos=49152
mget: test-download.data: file size decreased during transfer
{code}
> SFTP - Some files truncated on retrieval/download
> -
>
> Key: SSHD-725
> URL: https://issues.apache.org/jira/browse/SSHD-725
> Project: MINA SSHD
> Issue Type: Bug
>Affects Versions: 1.0.0, 1.2.0, 1.3.0
> Environment: Various flavors of Linux (ubuntu xenial, centos 6, AWS
> AMI 2016.x)
>Reporter: Russ Tennant
>Assignee: Goldstein Lyor
> Attachments: download-lftp-script-output-trace.txt, test-download.data
>
>
> When retrieving files (depending on file size), the files are truncated. We
> have also experienced a similar problem on file upload where we get extra
> data repeated from the last packet received.
> I don't have a test-case for the file upload issue - yet. Still trying to
> find the magic file size.
> I created a github project that demonstrates the truncation issue on
> retrieval.
> Project: https://github.com/VentureTech/sshd-test
> Example lftp Script:
> https://github.com/VentureTech/sshd-test/blob/master/download-lftp-script.txt
> Example lftp Script Output:
> https://github.com/VentureTech/sshd-test/blob/master/download-lftp-script-output.txt
> I used lftp to demonstrate the issue, but it happens with all clients I've
> tried (openssh, Intellij IDEA, kde's sftp ioslave).
> There is a gradle task lftpDownload that will start the server and run the
> lftp script.
> {code}./gradlew lftpDownload{code}
> Attached is the data file I used to trigger the issue.
> In the referenced example output lftp terminates the retrieval with the
> following
> {code}
> [116371] 2017-01-16 15:27:32 localhost <--- got a packet, length=120,
> type=101(STATUS), id=12
> [116371] 2017-01-16 15:27:32 localhost status code=1(EOF),
> message=Internal EOFException: Unable to read 16384 bytes from offset=65536
> of 5c18fc884da4646cb80a4b12519e6f16
> [116371] 2017-01-16 15:27:32 localhost eof
> [116371] 2017-01-16 15:27:32 ---> sending a packet, length=41, type=4(CLOSE),
> id=15
> [116371] 2017-01-16 15:27:32 expected pos=61564, actual pos=49152
> mget: test-download.data: file size decreased during transfer
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (SSHD-725) SFTP - Some files truncated on retrieval/download
[
https://issues.apache.org/jira/browse/SSHD-725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15954084#comment-15954084
]
Russ Tennant commented on SSHD-725:
---
[~lgoldstein] I tried all 4 changes you mentioned in the comment and the
problem still occurs. I'll try the 1.4 branch you mentioned next.
> SFTP - Some files truncated on retrieval/download
> -
>
> Key: SSHD-725
> URL: https://issues.apache.org/jira/browse/SSHD-725
> Project: MINA SSHD
> Issue Type: Bug
>Affects Versions: 1.0.0, 1.2.0, 1.3.0
> Environment: Various flavors of Linux (ubuntu xenial, centos 6, AWS
> AMI 2016.x)
>Reporter: Russ Tennant
>Assignee: Goldstein Lyor
> Attachments: download-lftp-script-output-trace.txt, test-download.data
>
>
> When retrieving files (depending on file size), the files are truncated. We
> have also experienced a similar problem on file upload where we get extra
> data repeated from the last packet received.
> I don't have a test-case for the file upload issue - yet. Still trying to
> find the magic file size.
> I created a github project that demonstrates the truncation issue on
> retrieval.
> Project: https://github.com/VentureTech/sshd-test
> Example lftp Script:
> https://github.com/VentureTech/sshd-test/blob/master/download-lftp-script.txt
> Example lftp Script Output:
> https://github.com/VentureTech/sshd-test/blob/master/download-lftp-script-output.txt
> I used lftp to demonstrate the issue, but it happens with all clients I've
> tried (openssh, Intellij IDEA, kde's sftp ioslave).
> There is a gradle task lftpDownload that will start the server and run the
> lftp script.
> {code}./gradlew lftpDownload{code}
> Attached is the data file I used to trigger the issue.
> In the referenced example output lftp terminates the retrieval with the
> following
> {code}
> [116371] 2017-01-16 15:27:32 localhost <--- got a packet, length=120,
> type=101(STATUS), id=12
> [116371] 2017-01-16 15:27:32 localhost status code=1(EOF),
> message=Internal EOFException: Unable to read 16384 bytes from offset=65536
> of 5c18fc884da4646cb80a4b12519e6f16
> [116371] 2017-01-16 15:27:32 localhost eof
> [116371] 2017-01-16 15:27:32 ---> sending a packet, length=41, type=4(CLOSE),
> id=15
> [116371] 2017-01-16 15:27:32 expected pos=61564, actual pos=49152
> mget: test-download.data: file size decreased during transfer
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (SSHD-725) SFTP - Some files truncated on retrieval/download
[
https://issues.apache.org/jira/browse/SSHD-725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15922755#comment-15922755
]
Russ Tennant commented on SSHD-725:
---
Sorry, I didn't see your comment until now. I must have my email misconfigured.
I'll try it out this week. Thanks.
> SFTP - Some files truncated on retrieval/download
> -
>
> Key: SSHD-725
> URL: https://issues.apache.org/jira/browse/SSHD-725
> Project: MINA SSHD
> Issue Type: Bug
>Affects Versions: 1.0.0, 1.2.0, 1.3.0
> Environment: Various flavors of Linux (ubuntu xenial, centos 6, AWS
> AMI 2016.x)
>Reporter: Russ Tennant
>Assignee: Goldstein Lyor
> Attachments: download-lftp-script-output-trace.txt, test-download.data
>
>
> When retrieving files (depending on file size), the files are truncated. We
> have also experienced a similar problem on file upload where we get extra
> data repeated from the last packet received.
> I don't have a test-case for the file upload issue - yet. Still trying to
> find the magic file size.
> I created a github project that demonstrates the truncation issue on
> retrieval.
> Project: https://github.com/VentureTech/sshd-test
> Example lftp Script:
> https://github.com/VentureTech/sshd-test/blob/master/download-lftp-script.txt
> Example lftp Script Output:
> https://github.com/VentureTech/sshd-test/blob/master/download-lftp-script-output.txt
> I used lftp to demonstrate the issue, but it happens with all clients I've
> tried (openssh, Intellij IDEA, kde's sftp ioslave).
> There is a gradle task lftpDownload that will start the server and run the
> lftp script.
> {code}./gradlew lftpDownload{code}
> Attached is the data file I used to trigger the issue.
> In the referenced example output lftp terminates the retrieval with the
> following
> {code}
> [116371] 2017-01-16 15:27:32 localhost <--- got a packet, length=120,
> type=101(STATUS), id=12
> [116371] 2017-01-16 15:27:32 localhost status code=1(EOF),
> message=Internal EOFException: Unable to read 16384 bytes from offset=65536
> of 5c18fc884da4646cb80a4b12519e6f16
> [116371] 2017-01-16 15:27:32 localhost eof
> [116371] 2017-01-16 15:27:32 ---> sending a packet, length=41, type=4(CLOSE),
> id=15
> [116371] 2017-01-16 15:27:32 expected pos=61564, actual pos=49152
> mget: test-download.data: file size decreased during transfer
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (SSHD-725) SFTP - Some files truncated on retrieval/download
[
https://issues.apache.org/jira/browse/SSHD-725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15826529#comment-15826529
]
Goldstein Lyor commented on SSHD-725:
-
While I did not find anything out of the ordinary, I have made some changes in
1.4 code that might help with this issue. Please clone the code at [this
branch|https://github.com/lgoldstein/mina-sshd/tree/SSHD-725], compile it and
try again.
To that effect, a few remarks regarding the test code you published - in the
interest of diagnosing the problem.
* Use {{SshServer.setupDefaultServer()}} instead of the builder in order to
obtain an initial server instance
* Don't call {{sshd.setSessionFactory(...)}}
* Try re-producing the issue (if it still occurs after my changes) without
setting the compression factories - perhaps the problem is related to
compression
* Try re-producing the issue without setting the
{{SftpSubsystem.MAX_PACKET_LENGTH_PROP}} value.
While there does not seem to be anything wrong with the test code you wrote, I
would like to minimize as much as possible "white noise" that might be the
result of some unforeseen side effect (which might still be a bug, but let's
deal with one problem at a time).
Let me know how it went.
> SFTP - Some files truncated on retrieval/download
> -
>
> Key: SSHD-725
> URL: https://issues.apache.org/jira/browse/SSHD-725
> Project: MINA SSHD
> Issue Type: Bug
>Affects Versions: 1.0.0, 1.2.0, 1.3.0
> Environment: Various flavors of Linux (ubuntu xenial, centos 6, AWS
> AMI 2016.x)
>Reporter: Russ Tennant
>Assignee: Goldstein Lyor
> Attachments: download-lftp-script-output-trace.txt, test-download.data
>
>
> When retrieving files (depending on file size), the files are truncated. We
> have also experienced a similar problem on file upload where we get extra
> data repeated from the last packet received.
> I don't have a test-case for the file upload issue - yet. Still trying to
> find the magic file size.
> I created a github project that demonstrates the truncation issue on
> retrieval.
> Project: https://github.com/VentureTech/sshd-test
> Example lftp Script:
> https://github.com/VentureTech/sshd-test/blob/master/download-lftp-script.txt
> Example lftp Script Output:
> https://github.com/VentureTech/sshd-test/blob/master/download-lftp-script-output.txt
> I used lftp to demonstrate the issue, but it happens with all clients I've
> tried (openssh, Intellij IDEA, kde's sftp ioslave).
> There is a gradle task lftpDownload that will start the server and run the
> lftp script.
> {code}./gradlew lftpDownload{code}
> Attached is the data file I used to trigger the issue.
> In the referenced example output lftp terminates the retrieval with the
> following
> {code}
> [116371] 2017-01-16 15:27:32 localhost <--- got a packet, length=120,
> type=101(STATUS), id=12
> [116371] 2017-01-16 15:27:32 localhost status code=1(EOF),
> message=Internal EOFException: Unable to read 16384 bytes from offset=65536
> of 5c18fc884da4646cb80a4b12519e6f16
> [116371] 2017-01-16 15:27:32 localhost eof
> [116371] 2017-01-16 15:27:32 ---> sending a packet, length=41, type=4(CLOSE),
> id=15
> [116371] 2017-01-16 15:27:32 expected pos=61564, actual pos=49152
> mget: test-download.data: file size decreased during transfer
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
