Protocol plugins should report progress to the fetcher
------------------------------------------------------
Key: NUTCH-569
URL: https://issues.apache.org/jira/browse/NUTCH-569
Project: Nutch
Issue Type: Improvement
Components: fetcher
Affects Versions: 1.0.0
Reporter: Andrzej Bialecki
Priority: Minor
When downloading very large files over slow connections, protocol plugins spend
long time in Protocol.getProtocolOutput(...). This sometimes leads to a timeout
in Fetcher / Fetcher2, with the message "aborting with hung threads". Protocol
plugins should periodically notify their caller about progress. In a situation
when the call to getProtocolOutput takes very long time to return, this will
help the caller to determine whether the wait is justified.
Preferably, the callback interface should allow the monitoring of not only the
binary progress / no-progress, but also the download speed, so that the caller
could terminate slow connections. E.g.
{noformat}
interface ProtocolReporter {
void progress(long bytesDownloaded);
}
{noformat}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
