[
https://issues.apache.org/jira/browse/HADOOP-2926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12574747#action_12574747
]
Tsz Wo (Nicholas), SZE commented on HADOOP-2926:
------------------------------------------------
> Shouldn't we try to make this idiom work well with HDFS?
This idiom is not obvious for multiple IOs. For example, the following codes
cannot handle exceptions correctly:
{code}
OutputStream out1 = fs.open(...);
OutputStream out2 = fs.open(...);
try {
out1.write(...);
out2.write(...);
} finally {
out1.close();
out2.close(); //not called if the previous line throws an exception
}
{code}
We still need something like IOUtils.closeStream to do try-catch if there are
more then one IOs.
I suggest we define the following method in IOUtils
{code}
public static void closeIO(Closeable... io) throws IOException {
List<IOException> ioexceptions = new ArrayList<IOException>();
for(Closeable c : io) {
try {io.close();}
catch(IOException e) {ioexceptions.add(e);}
}
if (!ioexceptions.isEmpty()) {
throw new IOException(...); //construct an IOException with the list
}
}
{code}
Then, multiple IOs can be closed together
{code}
OutputStream out1 = fs.open(...);
OutputStream out2 = fs.open(...);
try {
out1.write(...);
out2.write(...);
} finally {
IOUtils.closeIO(out1, out2);
}
{code}
> Ignoring IOExceptions on close
> ------------------------------
>
> Key: HADOOP-2926
> URL: https://issues.apache.org/jira/browse/HADOOP-2926
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.16.0
> Reporter: Owen O'Malley
> Assignee: dhruba borthakur
> Priority: Critical
> Fix For: 0.16.1
>
>
> Currently in HDFS there are a lot of calls to IOUtils.closeStream that are
> from finally blocks. I'm worried that this can lead to data corruption in the
> file system. Take the first instance in DataNode.copyBlock: it writes the
> block and then calls closeStream on the output stream. If there is an error
> at the end of the file that is detected in the close, it will be *completely*
> ignored. Note that logging the error is not enough, the error should be
> thrown so that the client knows the failure happened.
> {code}
> try {
> file1.write(...);
> file2.write(...);
> } finally {
> IOUtils.closeStream(file);
> }
> {code}
> is *bad*. It must be rewritten as:
> {code}
> try {
> file1.write(...);
> file2.write(...);
> file1.close(...);
> file2.close(...);
> } catch (IOException ie) {
> IOUtils.closeStream(file1);
> IOUtils.closeStream(file2);
> throw ie;
> }
> {code}
> I also think that IOUtils.closeStream should be renamed
> IOUtils.cleanupFailedStream or something to make it clear it can only be used
> after the write operation has failed and is being cleaned up.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
