[
https://issues.apache.org/jira/browse/MAPREDUCE-7108?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16506322#comment-16506322
]
Gera Shegalov edited comment on MAPREDUCE-7108 at 6/8/18 5:55 PM:
------------------------------------------------------------------
Thanks for the patch.
Instead of the manual a loop of closing Readers you can use
{{org.apache.hadoop.io.IOUtils#closeStreams}}}.
Since you are already changing this block, I think you can remove the catch
block
{code:java}
} catch (Exception e) {
fail("Fail to read from MapFileOutputFormat: " + e);
}{code}
because it does not look useful. And validation should be moved inside try as
well whereas fullyDelete be in finally. Would this make sense?
{code:java}
// Ensure getReaders call works and also ignores
// hidden filenames (_ or . prefixes)
MapFile.Reader[] readers = {};
try {
readers = getReaders(outDir, conf);
// validate output
validateMapFileOutputContent(FileSystem.get(job.getConfiguration()), outDir);
} finally {
IOUtils.closeStreams(r);
FileUtil.fullyDelete(new File(outDir.toString()));
} {code}
was (Author: jira.shegalov):
Thanks for the patch.
Instead of the manual a loop of closing Readers you can use
{{org.apache.hadoop.io.IOUtils#closeStreams}}}.
Since you are already changing this block, I think you can remove the catch
block
{code:java}
} catch (Exception e) {
fail("Fail to read from MapFileOutputFormat: " + e);
}{code}
because it does not look useful. And validation should be moved inside try as
well whereas fullyDelete be in finally. Would this make sense?
{code:java}
// Ensure getReaders call works and also ignores
// hidden filenames (_ or . prefixes)
MapFile.Reader[] readers = {};
try {
readers = getReaders(outDir, conf);
// validate output
validateMapFileOutputContent(FileSystem.get(job.getConfiguration()), outDir);
} finally {
IOUtils.closeStreams(r);
FileUtil.fullyDelete(new File(outDir.toString()));
}{code}
> TestFileOutputCommitter fails on Windows
> ----------------------------------------
>
> Key: MAPREDUCE-7108
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7108
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 3.1.0, 2.9.1
> Reporter: Zuoming Zhang
> Priority: Minor
> Labels: Windows
> Fix For: 3.1.0, 2.9.1
>
> Attachments: MAPREDUCE-7108-TestFileOutputCommitter-Report.000.txt,
> MAPREDUCE-7108.000.patch
>
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> TestFileOutputCommitter.java fails on Windows.
> Reason:
> This is because in testMapFileOutputCommitterInternal, after executing
> MapFileOutputFormat.getReaders(outDir, conf), the readers are not properly
> closed. So other apis that attempt to delete the existing file fails due to
> the file handler already being used.
>
> Solution:
> Close all readers after call MapFileOutputFormat.getReaders(outDir, conf);
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
