add progress callback feature to the slow FileUtil operations with ability to
cancel the work
---------------------------------------------------------------------------------------------
Key: HADOOP-5353
URL: https://issues.apache.org/jira/browse/HADOOP-5353
Project: Hadoop Core
Issue Type: Improvement
Components: fs
Affects Versions: 0.21.0
Reporter: Steve Loughran
Priority: Minor
This is something only of relevance of people doing front ends to FS
operations, and as they could take the code in FSUtil and add something with
this feature, its a blocker to none of them.
Current FileUtil.copy can take a long time to move large files around, but
there is no progress indicator to GUIs, or a way to cancel the operation
mid-way, short of interrupting the thread or closing the filesystem.
I propose a FileIOProgress interface to the copy ops, one that had a single
method to notify listeners of bytes read and written, and the number of files
handled.
{code}
interface FileIOProgress {
boolean progress(int files, long bytesRead, long bytesWritten);
}
The return value would be true to continue the operation, or false to stop the
copy and leave the FS in whatever incomplete state it is in currently.
it could even be fancier: have beginFileOperation and endFileOperation
callbacks to pass in the name of the current file being worked on, though I
don't have a personal need for that.
GUIs could show progress bars and cancel buttons, other tools could use the
interface to pass any cancellation notice upstream.
The FileUtil.copy operations would call this interface (blocking) after every
block copy, so the frequency of invocation would depend on block size and
network/disk speeds. Which is also why I don't propose having any percentage
done indicators; it's too hard to predict percentage of time done for
distributed file IO with any degree of accuracy.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
