[
https://issues.apache.org/jira/browse/HADOOP-1612?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12514554
]
Arun C Murthy commented on HADOOP-1612:
---------------------------------------
Christian, another pair of questions:
bq. In many of our applications we use the PIPES interface and create multiple
files using libhdfs in the output directory of the task.
I'd like to clarify that you are creating the files (using libhdfs) in the
temporary task sub-directory (i.e. ${map.output.dir}/_${taskid}), since only
files/dirs there can be safely relied upon to be promoted by the framework.
I'd also like to understand if you are using HADOOP-1576 which contained some
minor improvements to processing speculative tasks...
Thanks!
> listing of an output directory shortly after job completion fails
> -----------------------------------------------------------------
>
> Key: HADOOP-1612
> URL: https://issues.apache.org/jira/browse/HADOOP-1612
> Project: Hadoop
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.14.0
> Reporter: Christian Kunz
> Assignee: Arun C Murthy
> Priority: Blocker
> Fix For: 0.14.0
>
>
> Sometimes, after a job finishes, and another application wants to rename dfs
> files created by that job, listing of the output directory containing the
> newly created files fails. File creation and directory listing is done via
> libhdfs, but it is unlikely that this makes any difference, therefore, I add
> this to the mapred component.
> It might be a race condition: does the job complete before the files in the
> output directory are promoted?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
