Re: [net] FTPClient doesn't download full images
Hi,
I have no idea if this is related but I seem to remember from a life long, long
ago that you can set up and FTP connection in two modes ASCII and binary. The
default is (used to be) ASCII and when you feed binary data through that
connection then (some) non-ascii characters were translated.
Again, I may be talking complete nonsense here but it may be worth to check it
out.
Cheers,
Erwin
> On Jun 4, 2020, at 09:29, Julia Ruzicka wrote:
>
> @Robert Paasche
>
> The code is used in an AsyncTask and there's more error handling to it
> (including proper error messages,...), I just simplified it. ;)
>
> Have you ever tested it with images too? Text files always seem to download
> properly (no matter what size), it's just the images that are messed up.
>
> Thanks for the code snippet, I just ran my app with it multiple times and the
> files are all the right size, even the images, but no program is able to
> actually open them, so not sure what's going on with that now:
>
> if(singleFile.length > 0) { //check if file exists
>String localPath = localFolder + File.separator + filename;
>FTPFile single = singleFile[0];
>InputStream inStream = ftpClient.retrieveFileStream(filename);
>ByteArrayOutputStream buffer = new ByteArrayOutputStream();
>
>int nRead;
>byte[] data = new byte[(int) (single.getSize())]; //length was "*2" but
> why?
>
>while ((nRead = inStream.read(data, 0, data.length)) != -1) {
>buffer.write(data, 0, nRead);
>}
>
>buffer.flush();
>inStream.close();
>
>FileOutputStream fos = new FileOutputStream(localPath);
>fos.write(data);
>fos.flush();
>fos.close();
>
>if(ftpClient.completePendingCommand()) {
>Log.d(TAG,"DONE");
>} else {
>Log.d(TAG,"NOT DONE");
>}
> }
>
>
>
> -
> To unsubscribe, e-mail: user-unsubscr...@commons.apache.org
> For additional commands, e-mail: user-h...@commons.apache.org
>
-
To unsubscribe, e-mail: user-unsubscr...@commons.apache.org
For additional commands, e-mail: user-h...@commons.apache.org
Re: Re: [net] FTPClient doesn't download full images
No never used with images.
//length was "*2" but why?
That is for an buggy ftp-server, which mishandled multibyte encodings. On
UTF-16 Files he has reported the number of chars instead of bytes, so the
file size was just the half erverytime.
fos.write(data);
this should be something like:
fos.write(buffer.toByteArray());
Am Do., 4. Juni 2020 um 15:29 Uhr schrieb Julia Ruzicka <
julia.ruzi...@simutech.at>:
> @Robert Paasche
>
> The code is used in an AsyncTask and there's more error handling to it
> (including proper error messages,...), I just simplified it. ;)
>
> Have you ever tested it with images too? Text files always seem to
> download properly (no matter what size), it's just the images that are
> messed up.
>
> Thanks for the code snippet, I just ran my app with it multiple times and
> the files are all the right size, even the images, but no program is able
> to actually open them, so not sure what's going on with that now:
>
> if(singleFile.length > 0) { //check if file exists
> String localPath = localFolder + File.separator + filename;
> FTPFile single = singleFile[0];
> InputStream inStream = ftpClient.retrieveFileStream(filename);
> ByteArrayOutputStream buffer = new ByteArrayOutputStream();
>
> int nRead;
> byte[] data = new byte[(int) (single.getSize())]; //length was "*2"
> but why?
>
> while ((nRead = inStream.read(data, 0, data.length)) != -1) {
> buffer.write(data, 0, nRead);
> }
>
> buffer.flush();
> inStream.close();
>
> FileOutputStream fos = new FileOutputStream(localPath);
> fos.write(data);
> fos.flush();
> fos.close();
>
> if(ftpClient.completePendingCommand()) {
> Log.d(TAG,"DONE");
> } else {
> Log.d(TAG,"NOT DONE");
> }
> }
>
>
>
> -
> To unsubscribe, e-mail: user-unsubscr...@commons.apache.org
> For additional commands, e-mail: user-h...@commons.apache.org
>
>
Re: Re: [net] FTPClient doesn't download full images
@Robert Paasche
The code is used in an AsyncTask and there's more error handling to it
(including proper error messages,...), I just simplified it. ;)
Have you ever tested it with images too? Text files always seem to download
properly (no matter what size), it's just the images that are messed up.
Thanks for the code snippet, I just ran my app with it multiple times and the
files are all the right size, even the images, but no program is able to
actually open them, so not sure what's going on with that now:
if(singleFile.length > 0) { //check if file exists
String localPath = localFolder + File.separator + filename;
FTPFile single = singleFile[0];
InputStream inStream = ftpClient.retrieveFileStream(filename);
ByteArrayOutputStream buffer = new ByteArrayOutputStream();
int nRead;
byte[] data = new byte[(int) (single.getSize())]; //length was "*2" but why?
while ((nRead = inStream.read(data, 0, data.length)) != -1) {
buffer.write(data, 0, nRead);
}
buffer.flush();
inStream.close();
FileOutputStream fos = new FileOutputStream(localPath);
fos.write(data);
fos.flush();
fos.close();
if(ftpClient.completePendingCommand()) {
Log.d(TAG,"DONE");
} else {
Log.d(TAG,"NOT DONE");
}
}
-
To unsubscribe, e-mail: user-unsubscr...@commons.apache.org
For additional commands, e-mail: user-h...@commons.apache.org
AW: [net] FTPClient doesn't download full images
Hi everyone! I created a small test app that uses my "download" class on a button press (but doesn't do anything else) and did some more testing with it (with "FTP.BINARY_FILE_TYPE"): - Downloading a 14mb txt file (and its copies) always finishes with a complete file (even with a long file name) - Downloading a 12kb jpg (and its copies) always finishes with a complete file - Downloading the 8mb (png) or 6mb (jpg version) images sometimes completes but most of the time it doesn't - the file size is now always the same but there's still a black bar at the bottom of the images (height differs) - I uploaded a couple of random google images to my FTP server and most of them are affected by this problem too. You can find all the tested google images (including the sources) and the results in this imgur album: https://imgur.com/a/Mzngp11 I uploaded a couple of the images to a public FTP test server (https://dlptest.com/ftp-test/) and I get the same results with it. Has anyone got an idea what's wrong and how do fix this? - To unsubscribe, e-mail: user-unsubscr...@commons.apache.org For additional commands, e-mail: user-h...@commons.apache.org
Re: [net] FTPClient doesn't download full images
The following line is wrong:
while ((len = bis.read(buf)) > 0) { //Same result using only 'is'
read == -1 marks the end of the stream. 0 shows only that the stream is
still waiting for the next chunk of data.
Please read the JavaDocs:
> Returns:
> the number of bytes read, or -1 if the end of the stream has been reached.
while ((len = bis.read(buf)) >= 0) { //Same result using only 'is'
if (len >0)
out.write(buf, 0, len);
else
// prevent high CPU load
LockSupport.parkNanos(TimeUnit.MILLISECONDS.toNanos(500));
}
Re: [net] FTPClient doesn't download full images
Thanks for your reply!
I changed the code and did the usual AndroidStudio restart/clean/build but most
images are still downloaded incompletely.
I don't know if my other answer (sent about an hour ago) came through (I did
some more testing today), so here's an imgur album with test images and their
results:
https://imgur.com/a/Mzngp11
The last image (the satellite one) was now also affected for the first time.
-Ursprüngliche Nachricht-
Von: Robert Paasche
Gesendet: Donnerstag, 4. Juni 2020 12:18
An: Commons Users List
Betreff: Re: [net] FTPClient doesn't download full images
The following line is wrong:
while ((len = bis.read(buf)) > 0) { //Same result using only 'is'
read == -1 marks the end of the stream. 0 shows only that the stream is still
waiting for the next chunk of data.
Please read the JavaDocs:
> Returns:
> the number of bytes read, or -1 if the end of the stream has been reached.
while ((len = bis.read(buf)) >= 0) { //Same result using only 'is'
if (len >0)
out.write(buf, 0, len);
else
// prevent high CPU load
LockSupport.parkNanos(TimeUnit.MILLISECONDS.toNanos(500));
}
-
To unsubscribe, e-mail: user-unsubscr...@commons.apache.org
For additional commands, e-mail: user-h...@commons.apache.org
Re: [net] FTPClient doesn't download full images
Hmm maybe it is the bufferdStream.
That's working code I'm using in a project for years now:
private FTPClient createClient()
{
FTPClient ftp = new FTPClient();
FTPClientConfig config = new FTPClientConfig();
config.setServerTimeZoneId("UTC");
ftp.setControlKeepAliveTimeout(1000 * 60 * 5l);
ftp.setControlKeepAliveReplyTimeout(1000 * 60 * 5);
ftp.setDataTimeout(1000 * 60 * 5);
ftp.setRemoteVerificationEnabled(false);
return ftp;
}
if (FetchMode.BYTE_DATA.equals(fetchMode))
{
LOGGER.info("Using BINARY_FILE_TYPE.");
ftp.setFileType(FTP.BINARY_FILE_TYPE);
InputStream inStream = ftp.retrieveFileStream(ftpFile.getName());
ByteArrayOutputStream buffer = new ByteArrayOutputStream();
int nRead;
byte[] data = new byte[(int) (ftpFile.getSize() * 2)];
while ((nRead = inStream.read(data, 0, data.length)) != -1)
{
buffer.write(data, 0, nRead);
}
buffer.flush();
content.setByteData(buffer.toByteArray());
}
Re: [net] FTPClient doesn't download full images
If you want to read a stream without dealing with low-level headaches, you
can use
https://commons.apache.org/proper/commons-io/javadocs/api-release/org/apache/commons/io/IOUtils.html
Gary
On Thu, Jun 4, 2020 at 6:53 AM Julia Ruzicka
wrote:
> Thanks for your reply!
>
> I changed the code and did the usual AndroidStudio restart/clean/build but
> most images are still downloaded incompletely.
> I don't know if my other answer (sent about an hour ago) came through (I
> did some more testing today), so here's an imgur album with test images and
> their results:
> https://imgur.com/a/Mzngp11
>
> The last image (the satellite one) was now also affected for the first
> time.
>
> -Ursprüngliche Nachricht-
> Von: Robert Paasche
> Gesendet: Donnerstag, 4. Juni 2020 12:18
> An: Commons Users List
> Betreff: Re: [net] FTPClient doesn't download full images
>
> The following line is wrong:
> while ((len = bis.read(buf)) > 0) { //Same result using only 'is'
>
> read == -1 marks the end of the stream. 0 shows only that the stream is
> still waiting for the next chunk of data.
>
> Please read the JavaDocs:
>
> > Returns:
> > the number of bytes read, or -1 if the end of the stream has been
> reached.
>
>
> while ((len = bis.read(buf)) >= 0) { //Same result using only 'is'
> if (len >0)
>out.write(buf, 0, len);
> else
> // prevent high CPU load
> LockSupport.parkNanos(TimeUnit.MILLISECONDS.toNanos(500));
> }
>
>
>
> -
> To unsubscribe, e-mail: user-unsubscr...@commons.apache.org
> For additional commands, e-mail: user-h...@commons.apache.org
>
>
Re: [net] FTPClient doesn't download full images
Don't know if the code you have showed is your original one but in general its not a good idea to "catch and ignore" exceptions, yo either: - simply buble up to the caller - wrap into other exceptions if required by the API - explicitly handle/document I'm using comons-ftp for large files and small files and it always has worked quite well. Am 04.06.20 um 14:10 schrieb Julia Ruzicka: @Robert Paasche At first I tried to do it with a `FileOutputStream` and `ftpClient.retrieveFile(fileName,fos)` but that's how I noticed the problem. I then switched to using ` InputStream is = ftpClient.retrieveFileStream(filename)` and when that didn't work properly either, I added ` BufferedInputStream bis = new BufferedInputStream(is)`. What type of files are you downloading? What's your `content` and how do you write the data? With a DataOutputStream? - To unsubscribe, e-mail: user-unsubscr...@commons.apache.org For additional commands, e-mail: user-h...@commons.apache.org - To unsubscribe, e-mail: user-unsubscr...@commons.apache.org For additional commands, e-mail: user-h...@commons.apache.org
Re: Re: [net] FTPClient doesn't download full images
@ julia.ruzicka content is just a simple POJO holding the byte and meta data (like timestamps). In some situations I'm using a fileoutputstream after disconnecting from ftp. I'm downloading large (>300 mb) zip-Files. Am Do., 4. Juni 2020 um 14:10 Uhr schrieb Julia Ruzicka < julia.ruzi...@simutech.at>: > @Robert Paasche > > At first I tried to do it with a `FileOutputStream` and > `ftpClient.retrieveFile(fileName,fos)` but that's how I noticed the > problem. I then switched to using ` InputStream is = > ftpClient.retrieveFileStream(filename)` and when that didn't work properly > either, I added ` BufferedInputStream bis = new BufferedInputStream(is)`. > > What type of files are you downloading? > > What's your `content` and how do you write the data? With a > DataOutputStream? > > > > - > To unsubscribe, e-mail: user-unsubscr...@commons.apache.org > For additional commands, e-mail: user-h...@commons.apache.org > >
Re: Re: [net] FTPClient doesn't download full images
@Robert Paasche At first I tried to do it with a `FileOutputStream` and `ftpClient.retrieveFile(fileName,fos)` but that's how I noticed the problem. I then switched to using ` InputStream is = ftpClient.retrieveFileStream(filename)` and when that didn't work properly either, I added ` BufferedInputStream bis = new BufferedInputStream(is)`. What type of files are you downloading? What's your `content` and how do you write the data? With a DataOutputStream? - To unsubscribe, e-mail: user-unsubscr...@commons.apache.org For additional commands, e-mail: user-h...@commons.apache.org
