[jira] [Commented] (NET-686) Most files aren't downloaded completely from an FTP server
[ https://issues.apache.org/jira/browse/NET-686?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17244833#comment-17244833 ] Vitaliy Khudenko commented on NET-686: -- Please disregard my complain. Verified it from an Android emulator with public FTPS (test.rebex.net:21), obviously no VPN: no issues. Verified it from a PC console app to my FTPS + VPN: no issues. Verified it from a PC the Android real device to my FTPS + VPN: no issues. Fails only on the *Android emulator + VPN*. > Most files aren't downloaded completely from an FTP server > -- > > Key: NET-686 > URL: https://issues.apache.org/jira/browse/NET-686 > Project: Commons Net > Issue Type: Bug > Components: FTP >Affects Versions: 3.6, 3.7.2 > Environment: Win 10 > Java 8 > Android Studio 3.6.1 (min SDK 24, target SDK 27) >Reporter: JRRR >Priority: Major > Attachments: 2a-original.png, 2b-corrupt.png, 2c-corrupt.png, > 5a-original.jpg, 5b-corrupt.jpg, 5c-corrupt.jpg, DownloadProblem.java > > > About a month ago I opened another > [issue|https://issues.apache.org/jira/browse/NET-684] that was closed because > it wasn't reproducible with macOS and a public FTP server. > Short summary: Downloading files from an FTP server results in these files > randomly missing bytes. It looks like the download always "completes" and > there are no error messages/exceptions but random bytes in random files are > simply skipped. Images (jpg & png) are usually affected more (up to 30, maybe > 40, bytes smaller than the original), and are then also visibly corrupt, than > text files (usually only 2-3 bytes smaller, rarely more). > I'm working on an Android app (Win 10, Java 8, Android Studio 3.6.1, min SDK > 24, target SDK 27), which I'm testing with FTP servers in the same network > (1x Win 10, 1x Linux, both accessed via IP - "10.1.1.xxx"). No matter what > method in the library I use (retrieveFile, retrieveFileStream, > sendCommand(FTPCmd.RETRIEVE, filename)), most of the time there's at least a > single file that's corrupted. > I also tested the same code with public servers and even though I didn't have > a lot of time because those servers regularely delete uploaded files, I never > experienced said problem with them. > I even wrote my own mini-library (just for login/logout and download) using > Java's default "Socket" but I still had the same problem on Android Studio's > simulator/a real device. BUT: When I used the same code to create a small > Windows/Swing/Java app, there were no more corrupted files. > It looks like this bug is only affecting a very specific combination of > OS,...: > Android (emulator/real device) + Java (8) + FTP server in the same network > (accessed via IP) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (NET-686) Most files aren't downloaded completely from an FTP server
[ https://issues.apache.org/jira/browse/NET-686?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17244507#comment-17244507 ] Gary D. Gregory commented on NET-686: - Hi [~vkhudenko] I'm afraid I can't do much for you here unless you can provide a test in a PR or you can talk through which code in NET is wrong. This is challenging I know, since you are dealing with a whole software stack involving Android. See my my previous comment https://issues.apache.org/jira/browse/NET-686?focusedCommentId=17221564=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17221564 > Most files aren't downloaded completely from an FTP server > -- > > Key: NET-686 > URL: https://issues.apache.org/jira/browse/NET-686 > Project: Commons Net > Issue Type: Bug > Components: FTP >Affects Versions: 3.6, 3.7.2 > Environment: Win 10 > Java 8 > Android Studio 3.6.1 (min SDK 24, target SDK 27) >Reporter: JRRR >Priority: Major > Attachments: 2a-original.png, 2b-corrupt.png, 2c-corrupt.png, > 5a-original.jpg, 5b-corrupt.jpg, 5c-corrupt.jpg, DownloadProblem.java > > > About a month ago I opened another > [issue|https://issues.apache.org/jira/browse/NET-684] that was closed because > it wasn't reproducible with macOS and a public FTP server. > Short summary: Downloading files from an FTP server results in these files > randomly missing bytes. It looks like the download always "completes" and > there are no error messages/exceptions but random bytes in random files are > simply skipped. Images (jpg & png) are usually affected more (up to 30, maybe > 40, bytes smaller than the original), and are then also visibly corrupt, than > text files (usually only 2-3 bytes smaller, rarely more). > I'm working on an Android app (Win 10, Java 8, Android Studio 3.6.1, min SDK > 24, target SDK 27), which I'm testing with FTP servers in the same network > (1x Win 10, 1x Linux, both accessed via IP - "10.1.1.xxx"). No matter what > method in the library I use (retrieveFile, retrieveFileStream, > sendCommand(FTPCmd.RETRIEVE, filename)), most of the time there's at least a > single file that's corrupted. > I also tested the same code with public servers and even though I didn't have > a lot of time because those servers regularely delete uploaded files, I never > experienced said problem with them. > I even wrote my own mini-library (just for login/logout and download) using > Java's default "Socket" but I still had the same problem on Android Studio's > simulator/a real device. BUT: When I used the same code to create a small > Windows/Swing/Java app, there were no more corrupted files. > It looks like this bug is only affecting a very specific combination of > OS,...: > Android (emulator/real device) + Java (8) + FTP server in the same network > (accessed via IP) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (NET-686) Most files aren't downloaded completely from an FTP server
[
https://issues.apache.org/jira/browse/NET-686?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17244465#comment-17244465
]
Vitaliy Khudenko commented on NET-686:
--
Looks like I've encountered this issue too.
My setup is an Android app as an FTPS client running on a physical device
(Google Nexus, Android 8.1) or an emulator (Android 10) from Android Studio
(Win 10). VPN.
On the FTPS server I have 3 zip files (1, 4 and 74 KBytes).
Small files (1 and 4 KB) are never causing troubles. However for the largest
one the issue happens with ~25-33% probability.
The exact file size is 75'743 bytes. But it happens that only 75'741 bytes or
75'740 bytes is downloaded (missing just a few bytes) with no errors reported
by {{FTPSClient}}.
Here is my code (Kotlin):
{code:java}
private val CONNECT_TIMEOUT_MILLIS = 120_000 // 2 minutes
private val SO_TIMEOUT_MILLIS = 120_000 // 2 minutes
private val BUFFER_SIZE = 1024 // 1 KB
val client: FTPSClient = FTPSClient("TLS", false)
client.connectTimeout = CONNECT_TIMEOUT_MILLIS
client.connect(config.hostname, config.port)
client.soTimeout = SO_TIMEOUT_MILLIS
if (!FTPReply.isPositiveCompletion(client.replyCode)) {
throw Exception("FTP server refused connection")
}
if (!client.login(username, password)) {
throw Exception("FTP server refused to authenticate $username")
}
FileOutputStream(localTargetFile).use { outputStream: FileOutputStream ->
val status = client.retrieveFile(ftpFilePath, outputStream)
outputStream.flush()
check(status) { "negative retrieveFile status" } // throws Exception if
status is false, but this is always true
}
{code}
I tried to mitigate this with a retry mechanism where I would resume
downloading for the number of the missing bytes only (e.g. the second attempt
only fetches the last missing few bytes). But it still results in a corrupted
zip file, despite the total number of bytes is correct. Looks like the first
pass/attempt already brings a corrupted chunk, so resumed download does not
help:
{code:java}
repeat(ATTEMPTS_TO_DOWNLOAD) {
if (localTargetFile.exists()) {
client.restartOffset = localTargetFile.length()
FileOutputStream(localTargetFile, true)
} else {
FileOutputStream(localTargetFile)
}.use { outputStream: FileOutputStream ->
val status = client.retrieveFile(ftpFilePath, outputStream)
outputStream.flush()
check(status) { "negative retrieveFile status" } // throws Exception if
status is false, but this is always true
}
}{code}
The odd things I noticed (which is maybe due to FTPS server misconfig/missetup,
but FTPS server setup is beyond my control):
* {{FTPClient.mlistFile()}} always returns {{null}}
* I am not able to browse the FTPS content via WinSCP
> Most files aren't downloaded completely from an FTP server
> --
>
> Key: NET-686
> URL: https://issues.apache.org/jira/browse/NET-686
> Project: Commons Net
> Issue Type: Bug
> Components: FTP
>Affects Versions: 3.6, 3.7.2
> Environment: Win 10
> Java 8
> Android Studio 3.6.1 (min SDK 24, target SDK 27)
>Reporter: JRRR
>Priority: Major
> Attachments: 2a-original.png, 2b-corrupt.png, 2c-corrupt.png,
> 5a-original.jpg, 5b-corrupt.jpg, 5c-corrupt.jpg, DownloadProblem.java
>
>
> About a month ago I opened another
> [issue|https://issues.apache.org/jira/browse/NET-684] that was closed because
> it wasn't reproducible with macOS and a public FTP server.
> Short summary: Downloading files from an FTP server results in these files
> randomly missing bytes. It looks like the download always "completes" and
> there are no error messages/exceptions but random bytes in random files are
> simply skipped. Images (jpg & png) are usually affected more (up to 30, maybe
> 40, bytes smaller than the original), and are then also visibly corrupt, than
> text files (usually only 2-3 bytes smaller, rarely more).
> I'm working on an Android app (Win 10, Java 8, Android Studio 3.6.1, min SDK
> 24, target SDK 27), which I'm testing with FTP servers in the same network
> (1x Win 10, 1x Linux, both accessed via IP - "10.1.1.xxx"). No matter what
> method in the library I use (retrieveFile, retrieveFileStream,
> sendCommand(FTPCmd.RETRIEVE, filename)), most of the time there's at least a
> single file that's corrupted.
> I also tested the same code with public servers and even though I didn't have
> a lot of time because those servers regularely delete uploaded files, I never
> experienced said problem with them.
> I even wrote my own mini-library (just for login/logout and download) using
> Java's default "Socket" but I still had the same problem on Android Studio's
> simulator/a real device. BUT: When I used the same code to create a small
>
[jira] [Commented] (NET-686) Most files aren't downloaded completely from an FTP server
[ https://issues.apache.org/jira/browse/NET-686?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17222064#comment-17222064 ] JRRR commented on NET-686: -- [~ggregory] You mean, upload the file as new class to github? Sorry, I don't know how this whole pull/push github stuff works. :/ I did upload code for it here in August though, it's the code I was using when I noticed the problem. Since then I changed a couple of small things (like comparing the file size automatically,...) but the base code is still the same. > Most files aren't downloaded completely from an FTP server > -- > > Key: NET-686 > URL: https://issues.apache.org/jira/browse/NET-686 > Project: Commons Net > Issue Type: Bug > Components: FTP >Affects Versions: 3.6, 3.7.2 > Environment: Win 10 > Java 8 > Android Studio 3.6.1 (min SDK 24, target SDK 27) >Reporter: JRRR >Priority: Major > Attachments: 2a-original.png, 2b-corrupt.png, 2c-corrupt.png, > 5a-original.jpg, 5b-corrupt.jpg, 5c-corrupt.jpg, DownloadProblem.java > > > About a month ago I opened another > [issue|https://issues.apache.org/jira/browse/NET-684] that was closed because > it wasn't reproducible with macOS and a public FTP server. > Short summary: Downloading files from an FTP server results in these files > randomly missing bytes. It looks like the download always "completes" and > there are no error messages/exceptions but random bytes in random files are > simply skipped. Images (jpg & png) are usually affected more (up to 30, maybe > 40, bytes smaller than the original), and are then also visibly corrupt, than > text files (usually only 2-3 bytes smaller, rarely more). > I'm working on an Android app (Win 10, Java 8, Android Studio 3.6.1, min SDK > 24, target SDK 27), which I'm testing with FTP servers in the same network > (1x Win 10, 1x Linux, both accessed via IP - "10.1.1.xxx"). No matter what > method in the library I use (retrieveFile, retrieveFileStream, > sendCommand(FTPCmd.RETRIEVE, filename)), most of the time there's at least a > single file that's corrupted. > I also tested the same code with public servers and even though I didn't have > a lot of time because those servers regularely delete uploaded files, I never > experienced said problem with them. > I even wrote my own mini-library (just for login/logout and download) using > Java's default "Socket" but I still had the same problem on Android Studio's > simulator/a real device. BUT: When I used the same code to create a small > Windows/Swing/Java app, there were no more corrupted files. > It looks like this bug is only affecting a very specific combination of > OS,...: > Android (emulator/real device) + Java (8) + FTP server in the same network > (accessed via IP) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (NET-686) Most files aren't downloaded completely from an FTP server
[
https://issues.apache.org/jira/browse/NET-686?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17221564#comment-17221564
]
Gary D. Gregory commented on NET-686:
-
I mean create a test like {{org.apache.commons.net.ftp.FTPSClientTest}} that
demonstrates your issue.
> Most files aren't downloaded completely from an FTP server
> --
>
> Key: NET-686
> URL: https://issues.apache.org/jira/browse/NET-686
> Project: Commons Net
> Issue Type: Bug
> Components: FTP
>Affects Versions: 3.6, 3.7.2
> Environment: Win 10
> Java 8
> Android Studio 3.6.1 (min SDK 24, target SDK 27)
>Reporter: JRRR
>Priority: Major
> Attachments: 2a-original.png, 2b-corrupt.png, 2c-corrupt.png,
> 5a-original.jpg, 5b-corrupt.jpg, 5c-corrupt.jpg, DownloadProblem.java
>
>
> About a month ago I opened another
> [issue|https://issues.apache.org/jira/browse/NET-684] that was closed because
> it wasn't reproducible with macOS and a public FTP server.
> Short summary: Downloading files from an FTP server results in these files
> randomly missing bytes. It looks like the download always "completes" and
> there are no error messages/exceptions but random bytes in random files are
> simply skipped. Images (jpg & png) are usually affected more (up to 30, maybe
> 40, bytes smaller than the original), and are then also visibly corrupt, than
> text files (usually only 2-3 bytes smaller, rarely more).
> I'm working on an Android app (Win 10, Java 8, Android Studio 3.6.1, min SDK
> 24, target SDK 27), which I'm testing with FTP servers in the same network
> (1x Win 10, 1x Linux, both accessed via IP - "10.1.1.xxx"). No matter what
> method in the library I use (retrieveFile, retrieveFileStream,
> sendCommand(FTPCmd.RETRIEVE, filename)), most of the time there's at least a
> single file that's corrupted.
> I also tested the same code with public servers and even though I didn't have
> a lot of time because those servers regularely delete uploaded files, I never
> experienced said problem with them.
> I even wrote my own mini-library (just for login/logout and download) using
> Java's default "Socket" but I still had the same problem on Android Studio's
> simulator/a real device. BUT: When I used the same code to create a small
> Windows/Swing/Java app, there were no more corrupted files.
> It looks like this bug is only affecting a very specific combination of
> OS,...:
> Android (emulator/real device) + Java (8) + FTP server in the same network
> (accessed via IP)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (NET-686) Most files aren't downloaded completely from an FTP server
[ https://issues.apache.org/jira/browse/NET-686?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17221401#comment-17221401 ] JRRR commented on NET-686: -- Thanks for the prompt replies! @[~ggregory] Sorry, what do you mean exactly? @[~b.eckenfels] It might be a problem with maybe Android itself (the non-Apache socket code works fine in a normal Windows app) but the library doesn't do anything to work around it and the whole time I've been working on my app (probably more than a year by now) I haven't found a way around it either. The bug happens with both Android Studio's emulator and a real phone or tablet. All the devices are only "online" within the same network: The Emulator through the LAN connection of the PC, the phone via wifi but they don't actually need internet access, as both servers (1x Linux, 1x Windows) are within the same network too (I access them via "10.1.1.xxx"). I haven't found a proper public server that I can thoroughly test this with because the existing ones delete uploaded files either instantly or every 30 minutes. > Most files aren't downloaded completely from an FTP server > -- > > Key: NET-686 > URL: https://issues.apache.org/jira/browse/NET-686 > Project: Commons Net > Issue Type: Bug > Components: FTP >Affects Versions: 3.6, 3.7.2 > Environment: Win 10 > Java 8 > Android Studio 3.6.1 (min SDK 24, target SDK 27) >Reporter: JRRR >Priority: Major > Attachments: 2a-original.png, 2b-corrupt.png, 2c-corrupt.png, > 5a-original.jpg, 5b-corrupt.jpg, 5c-corrupt.jpg, DownloadProblem.java > > > About a month ago I opened another > [issue|https://issues.apache.org/jira/browse/NET-684] that was closed because > it wasn't reproducible with macOS and a public FTP server. > Short summary: Downloading files from an FTP server results in these files > randomly missing bytes. It looks like the download always "completes" and > there are no error messages/exceptions but random bytes in random files are > simply skipped. Images (jpg & png) are usually affected more (up to 30, maybe > 40, bytes smaller than the original), and are then also visibly corrupt, than > text files (usually only 2-3 bytes smaller, rarely more). > I'm working on an Android app (Win 10, Java 8, Android Studio 3.6.1, min SDK > 24, target SDK 27), which I'm testing with FTP servers in the same network > (1x Win 10, 1x Linux, both accessed via IP - "10.1.1.xxx"). No matter what > method in the library I use (retrieveFile, retrieveFileStream, > sendCommand(FTPCmd.RETRIEVE, filename)), most of the time there's at least a > single file that's corrupted. > I also tested the same code with public servers and even though I didn't have > a lot of time because those servers regularely delete uploaded files, I never > experienced said problem with them. > I even wrote my own mini-library (just for login/logout and download) using > Java's default "Socket" but I still had the same problem on Android Studio's > simulator/a real device. BUT: When I used the same code to create a small > Windows/Swing/Java app, there were no more corrupted files. > It looks like this bug is only affecting a very specific combination of > OS,...: > Android (emulator/real device) + Java (8) + FTP server in the same network > (accessed via IP) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (NET-686) Most files aren't downloaded completely from an FTP server
[ https://issues.apache.org/jira/browse/NET-686?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17221379#comment-17221379 ] Gary D. Gregory commented on NET-686: - I wonder if we could only make sense of this with an integration test in a PR. This would help rule out a problem with the current server being used. Take a look at the current tests. IIRC the FTPS test stands up a real FTP embedded server, so start there... > Most files aren't downloaded completely from an FTP server > -- > > Key: NET-686 > URL: https://issues.apache.org/jira/browse/NET-686 > Project: Commons Net > Issue Type: Bug > Components: FTP >Affects Versions: 3.6, 3.7.2 > Environment: Win 10 > Java 8 > Android Studio 3.6.1 (min SDK 24, target SDK 27) >Reporter: JRRR >Priority: Major > Attachments: 2a-original.png, 2b-corrupt.png, 2c-corrupt.png, > 5a-original.jpg, 5b-corrupt.jpg, 5c-corrupt.jpg, DownloadProblem.java > > > About a month ago I opened another > [issue|https://issues.apache.org/jira/browse/NET-684] that was closed because > it wasn't reproducible with macOS and a public FTP server. > Short summary: Downloading files from an FTP server results in these files > randomly missing bytes. It looks like the download always "completes" and > there are no error messages/exceptions but random bytes in random files are > simply skipped. Images (jpg & png) are usually affected more (up to 30, maybe > 40, bytes smaller than the original), and are then also visibly corrupt, than > text files (usually only 2-3 bytes smaller, rarely more). > I'm working on an Android app (Win 10, Java 8, Android Studio 3.6.1, min SDK > 24, target SDK 27), which I'm testing with FTP servers in the same network > (1x Win 10, 1x Linux, both accessed via IP - "10.1.1.xxx"). No matter what > method in the library I use (retrieveFile, retrieveFileStream, > sendCommand(FTPCmd.RETRIEVE, filename)), most of the time there's at least a > single file that's corrupted. > I also tested the same code with public servers and even though I didn't have > a lot of time because those servers regularely delete uploaded files, I never > experienced said problem with them. > I even wrote my own mini-library (just for login/logout and download) using > Java's default "Socket" but I still had the same problem on Android Studio's > simulator/a real device. BUT: When I used the same code to create a small > Windows/Swing/Java app, there were no more corrupted files. > It looks like this bug is only affecting a very specific combination of > OS,...: > Android (emulator/real device) + Java (8) + FTP server in the same network > (accessed via IP) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (NET-686) Most files aren't downloaded completely from an FTP server
[ https://issues.apache.org/jira/browse/NET-686?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17221380#comment-17221380 ] Bernd Eckenfels commented on NET-686: - Since you get the same problem with sockets I would say its not related to commons-net but a problem with your handy or handy network or server. Maybe using TLS will help? or a vpn? some providers really do strange things to mobile traffic. > Most files aren't downloaded completely from an FTP server > -- > > Key: NET-686 > URL: https://issues.apache.org/jira/browse/NET-686 > Project: Commons Net > Issue Type: Bug > Components: FTP >Affects Versions: 3.6, 3.7.2 > Environment: Win 10 > Java 8 > Android Studio 3.6.1 (min SDK 24, target SDK 27) >Reporter: JRRR >Priority: Major > Attachments: 2a-original.png, 2b-corrupt.png, 2c-corrupt.png, > 5a-original.jpg, 5b-corrupt.jpg, 5c-corrupt.jpg, DownloadProblem.java > > > About a month ago I opened another > [issue|https://issues.apache.org/jira/browse/NET-684] that was closed because > it wasn't reproducible with macOS and a public FTP server. > Short summary: Downloading files from an FTP server results in these files > randomly missing bytes. It looks like the download always "completes" and > there are no error messages/exceptions but random bytes in random files are > simply skipped. Images (jpg & png) are usually affected more (up to 30, maybe > 40, bytes smaller than the original), and are then also visibly corrupt, than > text files (usually only 2-3 bytes smaller, rarely more). > I'm working on an Android app (Win 10, Java 8, Android Studio 3.6.1, min SDK > 24, target SDK 27), which I'm testing with FTP servers in the same network > (1x Win 10, 1x Linux, both accessed via IP - "10.1.1.xxx"). No matter what > method in the library I use (retrieveFile, retrieveFileStream, > sendCommand(FTPCmd.RETRIEVE, filename)), most of the time there's at least a > single file that's corrupted. > I also tested the same code with public servers and even though I didn't have > a lot of time because those servers regularely delete uploaded files, I never > experienced said problem with them. > I even wrote my own mini-library (just for login/logout and download) using > Java's default "Socket" but I still had the same problem on Android Studio's > simulator/a real device. BUT: When I used the same code to create a small > Windows/Swing/Java app, there were no more corrupted files. > It looks like this bug is only affecting a very specific combination of > OS,...: > Android (emulator/real device) + Java (8) + FTP server in the same network > (accessed via IP) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (NET-686) Most files aren't downloaded completely from an FTP server
[ https://issues.apache.org/jira/browse/NET-686?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17221367#comment-17221367 ] JRRR commented on NET-686: -- I just upgraded to version 3.7.2, unfortunately the problem is still there. > Most files aren't downloaded completely from an FTP server > -- > > Key: NET-686 > URL: https://issues.apache.org/jira/browse/NET-686 > Project: Commons Net > Issue Type: Bug > Components: FTP >Affects Versions: 3.6, 3.7.2 > Environment: Win 10 > Java 8 > Android Studio 3.6.1 (min SDK 24, target SDK 27) >Reporter: JRRR >Priority: Major > Attachments: 2a-original.png, 2b-corrupt.png, 2c-corrupt.png, > 5a-original.jpg, 5b-corrupt.jpg, 5c-corrupt.jpg, DownloadProblem.java > > > About a month ago I opened another > [issue|https://issues.apache.org/jira/browse/NET-684] that was closed because > it wasn't reproducible with macOS and a public FTP server. > Short summary: Downloading files from an FTP server results in these files > randomly missing bytes. It looks like the download always "completes" and > there are no error messages/exceptions but random bytes in random files are > simply skipped. Images (jpg & png) are usually affected more (up to 30, maybe > 40, bytes smaller than the original), and are then also visibly corrupt, than > text files (usually only 2-3 bytes smaller, rarely more). > I'm working on an Android app (Win 10, Java 8, Android Studio 3.6.1, min SDK > 24, target SDK 27), which I'm testing with FTP servers in the same network > (1x Win 10, 1x Linux, both accessed via IP - "10.1.1.xxx"). No matter what > method in the library I use (retrieveFile, retrieveFileStream, > sendCommand(FTPCmd.RETRIEVE, filename)), most of the time there's at least a > single file that's corrupted. > I also tested the same code with public servers and even though I didn't have > a lot of time because those servers regularely delete uploaded files, I never > experienced said problem with them. > I even wrote my own mini-library (just for login/logout and download) using > Java's default "Socket" but I still had the same problem on Android Studio's > simulator/a real device. BUT: When I used the same code to create a small > Windows/Swing/Java app, there were no more corrupted files. > It looks like this bug is only affecting a very specific combination of > OS,...: > Android (emulator/real device) + Java (8) + FTP server in the same network > (accessed via IP) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (NET-686) Most files aren't downloaded completely from an FTP server
[ https://issues.apache.org/jira/browse/NET-686?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17183190#comment-17183190 ] JRRR commented on NET-686: -- Sorry for the late reply (vacation). There are simply bytes missing in seemingly random places (usually in the second half of the file), which completely "breaks" image files but doesn't have too much of an impact on text files (apart from content missing obviously). Some files do seem to break more often than others (e.g. example image "2a") but if I run the same download command with the same files multiple times, I'll always get different results: Not every file is corrupt every time it's downloaded, one time it's okay, then it's corrupt the next three or maybe even five times and when it is, the amount of missing bytes also varies. It does seem to happen more often with images than with txt files though and if I use a big byte buffer, it happens more often than with a small one. At one point I was also pretty sure that files downloaded from one of the servers resulted in more corrupted files (compared to the other one) and today the Windows server was the "problem child" but I can't remember if it was always that one. I uploaded examples (the original files I found on google) that I just downloaded: 4/4 images and 1/2 text files were corrupt. * "retrieveFileStream" version * Only `InputStream`/no `BufferedInputStream` * Big byte buffer (`singleFile[0]`.getSize()) * Moved "setFileType" A smaller buffer (5000) resulted in 2/4 corrupted images and 2/2 corrupted text files at first. The second time I ended up with 2/4 corrupted images and 0/2 corrupted text files. With a buffer size of 1000 all 6 files were okay but if I ran it multiple times, there'd still be a corrupt file at one point, so I can't just rely on that. > Most files aren't downloaded completely from an FTP server > -- > > Key: NET-686 > URL: https://issues.apache.org/jira/browse/NET-686 > Project: Commons Net > Issue Type: Bug > Components: FTP >Affects Versions: 3.6 > Environment: Win 10 > Java 8 > Android Studio 3.6.1 (min SDK 24, target SDK 27) >Reporter: JRRR >Priority: Major > Attachments: 2a-original.png, 2b-corrupt.png, 2c-corrupt.png, > 5a-original.jpg, 5b-corrupt.jpg, 5c-corrupt.jpg, DownloadProblem.java > > > About a month ago I opened another > [issue|https://issues.apache.org/jira/browse/NET-684] that was closed because > it wasn't reproducible with macOS and a public FTP server. > Short summary: Downloading files from an FTP server results in these files > randomly missing bytes. It looks like the download always "completes" and > there are no error messages/exceptions but random bytes in random files are > simply skipped. Images (jpg & png) are usually affected more (up to 30, maybe > 40, bytes smaller than the original), and are then also visibly corrupt, than > text files (usually only 2-3 bytes smaller, rarely more). > I'm working on an Android app (Win 10, Java 8, Android Studio 3.6.1, min SDK > 24, target SDK 27), which I'm testing with FTP servers in the same network > (1x Win 10, 1x Linux, both accessed via IP - "10.1.1.xxx"). No matter what > method in the library I use (retrieveFile, retrieveFileStream, > sendCommand(FTPCmd.RETRIEVE, filename)), most of the time there's at least a > single file that's corrupted. > I also tested the same code with public servers and even though I didn't have > a lot of time because those servers regularely delete uploaded files, I never > experienced said problem with them. > I even wrote my own mini-library (just for login/logout and download) using > Java's default "Socket" but I still had the same problem on Android Studio's > simulator/a real device. BUT: When I used the same code to create a small > Windows/Swing/Java app, there were no more corrupted files. > It looks like this bug is only affecting a very specific combination of > OS,...: > Android (emulator/real device) + Java (8) + FTP server in the same network > (accessed via IP) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (NET-686) Most files aren't downloaded completely from an FTP server
[ https://issues.apache.org/jira/browse/NET-686?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17169901#comment-17169901 ] Bernd Eckenfels commented on NET-686: - Can you provide a binary difference what exactly is wrong or provide a Testfile and it's corrupt result? Is the same test file always broken when you request it multiple times? Btw you don't need the buffered input stream if you read with larger buffer. I would move the setFileType before the retrieve (after the list files). > Most files aren't downloaded completely from an FTP server > -- > > Key: NET-686 > URL: https://issues.apache.org/jira/browse/NET-686 > Project: Commons Net > Issue Type: Bug > Components: FTP >Affects Versions: 3.6 > Environment: Win 10 > Java 8 > Android Studio 3.6.1 (min SDK 24, target SDK 27) >Reporter: JRRR >Priority: Major > Attachments: DownloadProblem.java > > > About a month ago I opened another > [issue|https://issues.apache.org/jira/browse/NET-684] that was closed because > it wasn't reproducible with macOS and a public FTP server. > Short summary: Downloading files from an FTP server results in these files > randomly missing bytes. It looks like the download always "completes" and > there are no error messages/exceptions but random bytes in random files are > simply skipped. Images (jpg & png) are usually affected more (up to 30, maybe > 40, bytes smaller than the original), and are then also visibly corrupt, than > text files (usually only 2-3 bytes smaller, rarely more). > I'm working on an Android app (Win 10, Java 8, Android Studio 3.6.1, min SDK > 24, target SDK 27), which I'm testing with FTP servers in the same network > (1x Win 10, 1x Linux, both accessed via IP - "10.1.1.xxx"). No matter what > method in the library I use (retrieveFile, retrieveFileStream, > sendCommand(FTPCmd.RETRIEVE, filename)), most of the time there's at least a > single file that's corrupted. > I also tested the same code with public servers and even though I didn't have > a lot of time because those servers regularely delete uploaded files, I never > experienced said problem with them. > I even wrote my own mini-library (just for login/logout and download) using > Java's default "Socket" but I still had the same problem on Android Studio's > simulator/a real device. BUT: When I used the same code to create a small > Windows/Swing/Java app, there were no more corrupted files. > It looks like this bug is only affecting a very specific combination of > OS,...: > Android (emulator/real device) + Java (8) + FTP server in the same network > (accessed via IP) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (NET-686) Most files aren't downloaded completely from an FTP server
[ https://issues.apache.org/jira/browse/NET-686?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17169852#comment-17169852 ] JRRR commented on NET-686: -- [~sebb] Sorry, I already changed that in my code but forgot to change it in the file too. > Most files aren't downloaded completely from an FTP server > -- > > Key: NET-686 > URL: https://issues.apache.org/jira/browse/NET-686 > Project: Commons Net > Issue Type: Bug > Components: FTP >Affects Versions: 3.6 > Environment: Win 10 > Java 8 > Android Studio 3.6.1 (min SDK 24, target SDK 27) >Reporter: JRRR >Priority: Major > Attachments: DownloadProblem.java > > > About a month ago I opened another > [issue|https://issues.apache.org/jira/browse/NET-684] that was closed because > it wasn't reproducible with macOS and a public FTP server. > Short summary: Downloading files from an FTP server results in these files > randomly missing bytes. It looks like the download always "completes" and > there are no error messages/exceptions but random bytes in random files are > simply skipped. Images (jpg & png) are usually affected more (up to 30, maybe > 40, bytes smaller than the original), and are then also visibly corrupt, than > text files (usually only 2-3 bytes smaller, rarely more). > I'm working on an Android app (Win 10, Java 8, Androis Studio 3.6.1, min SDK > 24, target SDK 27), which I'm testing with FTP servers in the same network > (1x Win 10, 1x Linux, both accessed via IP - "10.1.1.xxx"). No matter what > method in the library I use (retrieveFile, retrieveFileStream, > sendCommand(FTPCmd.RETRIEVE, filename)), most of the time there's at least a > single file that's corrupted. > I also tested the same code with public servers and even though I didn't have > a lot of time because those servers regularely delete uploaded files, I never > experienced said problem with them. > I even wrote my own mini-library (just for login/logout and download) using > Java's default "Socket" but I still had the same problem on Android Studio's > simulator/a real device. BUT: When I used the same code to create a small > Windows/Swing/Java app, there were no more corrupted files. > It looks like this bug is only affecting a very specific combination of > OS,...: > Android (emulator/real device) + Java (8) + FTP server in the same network > (accessed via IP) -- This message was sent by Atlassian Jira (v8.3.4#803005)
