I wrote a simple JUnit, did not stick deeper into it but at the first
moment, it behaves just like at my application. So i *guess* its the
same problem here:
package de.mydomain.jsfdl.app.sfdl;
import org.apache.commons.net.ftp.FTP;
import org.apache.commons.net.ftp.FTPClient;
import org.apache.commons.net.ftp.FTPFile;
import org.junit.Test;
import java.io.File;
import java.io.FileOutputStream;
import java.io.InputStream;
import java.util.concurrent.Executors;
import java.util.concurrent.ScheduledExecutorService;
public class SFDLFileDownloaderTest {
private final Stringhostname ="ftp.rz.uni-wuerzburg.de";
private final int port = FTP.DEFAULT_PORT;
private final Stringusername ="anonymous";
private final Stringpassword ="b...@somemail.de";
private final StringdirectoryOfInterest ="/pub/MIRROR/kde/4.14.0/src/";
private final StringlocalDestinationDir ="/home/oli/Downloads/test";
@Test public void testFTP()throws Exception {
final FTPClient ftpClient =new FTPClient();
ftpClient.connect(hostname,port);
System.out.println(ftpClient.getReplyString());
ftpClient.login(username,password);
ftpClient.enterLocalPassiveMode();
System.out.println(ftpClient.getReplyString());
ftpClient.setFileTransferMode(FTPClient.PASSIVE_LOCAL_DATA_CONNECTION_MODE);
System.out.println(ftpClient.getReplyString());
ftpClient.cwd(directoryOfInterest);
System.out.println(ftpClient.getReplyString());
FTPFile files[] = ftpClient.listFiles();
System.out.println(ftpClient.getReplyString());
for(FTPFile ftpFile : files) {
final ScheduledExecutorService scheduledExecutorService =
Executors.newScheduledThreadPool(5);
scheduledExecutorService.submit(new
DownloadRunnable(localDestinationDir, ftpFile,directoryOfInterest, ftpClient));
}
}
private class DownloadRunnableimplements Runnable {
private final StringdestDir;
private final FTPFileftpFile;
private final StringsourceDir;
private final FTPClientclient;
public DownloadRunnable(String destDir, FTPFile ftpFile, String
sourceDir, FTPClient client) {
this.destDir = destDir;
this.ftpFile = ftpFile;
this.sourceDir = sourceDir;
this.client = client;
}
@Override public void run() {
try {
new File(destDir).mkdirs();
final File outFile =new File(destDir,ftpFile.getName());
final FileOutputStream fos =new FileOutputStream(outFile);
final InputStream inputStream
=client.retrieveFileStream(sourceDir +ftpFile.getName());
byte b[] =new byte[4096];
int len;
while ((len = inputStream.read(b)) != -1) {
fos.write(b,0, len);
}
client.completePendingCommand();
fos.flush();
fos.close();
System.out.println(client.getReplyString());
}catch (Exception ex) {
System.out.println(client.getReplyString());
ex.printStackTrace();
}
}
}
}
I just read http://wiki.apache.org/commons/Net/FrequentlyAskedQuestions
*Q: Are the Commons-Net classes thread-safe? For example, can you have
multiple instances of FTPClient, each with its own thread running inside?*
*A:*Multiple instances of FTPClient can be used to connect to multiple
(or the same) FTP serverand concurrently retrieve files. If you want to
share a single FTPClient instance between multiplethreads, you must
serialize access to the object with critical sections.
Can someone tell me what is meant by "critical sections" ?
Am 30.08.2016 um 16:23 schrieb sebb:
On 29 August 2016 at 18:28, Oliver Zemann <oliver.zem...@gmail.com> wrote:
My download method looks like this:
private void downloadDirectory(FTPClient client, String sourceDir, String
destDir,
ScheduledExecutorService
scheduledExecutorService) throws IOException {
final FTPFile[] ftpFiles = client.listFiles(sourceDir);
for (final FTPFile ftpFile : ftpFiles) {
//skip . and ..
if (ftpFile.isDirectory() && (ftpFile.getName().equals(".") ||
ftpFile.getName().equals(".."))) {
continue;
}
if (ftpFile.isDirectory()) {
downloadDirectory(client, sourceDir + ftpFile.getName(),
destDir + "/" + ftpFile.getName(), scheduledExecutorService);
} else {
// scheduledExecutorService.submit(new
DownloadRunnable(destDir, ftpFile, sourceDir, client));
FileOutputStream fos = new
FileOutputStream(ftpFile.getName());
client.retrieveFile(sourceDir + ftpFile.getName(), fos );
fos.flush();
fos.close();
}
}
}
When i run it like this, it works and it downloads the files. But when i use
the scheduledExecuterService, it fails.
So i replaced the 2 lines with my download method (where it uses the
inputstream) to see what happens:
} else {
// scheduledExecutorService.submit(new
DownloadRunnable(destDir, ftpFile, sourceDir, client));
new File(destDir).mkdirs();
final File outFile = new File(destDir, ftpFile.getName());
final FileOutputStream fos = new FileOutputStream(outFile);
final InputStream inputStream =
client.retrieveFileStream(sourceDir + ftpFile.getName());
byte[] buffer = new byte[8096];
int len = -1;
try {
len = inputStream.read(buffer, 0, buffer.length);
} catch (Exception ex) {
ex.printStackTrace();
}
while (len != -1) {
fos.write(buffer, 0, len);
len = inputStream.read(buffer);
if (Thread.interrupted()) {
throw new InterruptedException();
}
}
client.completePendingCommand();
// informListenersWeFinishedOneFile(sourceDir +
ftpFile.getName(), bytesWritten);
fos.flush();
fos.close();
}
}
That works too?!
I will go deeper into that at weekend and check whats going on, but at the
moment i really dont understand it. Maybe some kind of concurrency problem
with FTPClient?
That seems the most likely.
FTPClient is not guaranteed to be thread-safe.
I'm not sure that the java.io classes such as InputStream are thread
safe either.
Am 29.08.2016 um 01:40 schrieb Martin Gainty:
possible timeout waiting for FTP to reply
examples.ftp.FTPClientExample says to increase FTP reply timeout with -w
parameter
if (args[base].equals("-w")) {
controlKeepAliveReplyTimeout = Integer.parseInt(args[++base]); }
or with FTPClient:ftpClient.setControlKeepAliveReplyTimeout(2000); //2 sec
reply timeout
?
Martin
______________________________________________
Date: Sun, 28 Aug 2016 20:06:46 +0200
From: e...@zusammenkunft.net
To: user@commons.apache.org
CC: oliver.zem...@gmail.com
Subject: Re: NPE in getRemoteAdress
Hello,
I am not sure about your NPE, but this code here ignores the result of
the read call. It cannot deal with short reads:
Am Sun, 28 Aug 2016
15:50:36 +0200 schrieb Oliver Zemann <oliver.zem...@gmail.com>:
byte b[] =new byte[4096];
while (inputStream.read(b) != -1) {
fos.write(b);
bytesWritten += b.length;
Gruss
Bernd
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscr...@commons.apache.org
For additional commands, e-mail: user-h...@commons.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscr...@commons.apache.org
For additional commands, e-mail: user-h...@commons.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscr...@commons.apache.org
For additional commands, e-mail: user-h...@commons.apache.org
