[
https://issues.apache.org/jira/browse/HADOOP-7973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13190080#comment-13190080
]
Daryn Sharp commented on HADOOP-7973:
-------------------------------------
bq. I guess you would need minidfs cluster instead of running miniMR on top of
local filesystem?
I wasn't aware the MR minicluster used local fs, I'll check that. The beauty
is if I've missed something, the tests will immediately fail.
bq. Also, for the actual change, is this sufficient to avoid the close failure?
I thought that FileSystem throwing the actual error is not from the MapTask
directly but the one opened inside TextInputFormat->LineRecordReader
The recorder reader is opened by {{MapTask#getSplitDetails}} in the framework.
It will now use a distinct fs object separate from any in the user task code,
just like it (accidentally) used to do. This is the stream that we are seeing
in exceptions.
bq. Does this apply to output stream as well? If it really does, then
something needs to be changed on the reducer side as well?
Yes, the collector gets a unique fs. The reducer writes the the raw local
filesystem.
> DistributedFileSystem close has severe consequences
> ---------------------------------------------------
>
> Key: HADOOP-7973
> URL: https://issues.apache.org/jira/browse/HADOOP-7973
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
> Affects Versions: 1.0.0
> Reporter: Daryn Sharp
> Assignee: Daryn Sharp
> Priority: Blocker
> Attachments: HADOOP-7973-2.patch, HADOOP-7973-3.patch,
> HADOOP-7973-4.patch, HADOOP-7973.patch
>
>
> The way {{FileSystem#close}} works is very problematic. Since the
> {{FileSystems}} are cached, any {{close}} by any caller will cause problems
> for every other reference to it. Will add more detail in the comments.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
