[ http://issues.apache.org/jira/browse/HADOOP-318?page=comments#action_12417539 ]
Milind Bhandarkar commented on HADOOP-318: ------------------------------------------ I don't see how this can be done without breaking backward-compatibility. Therefore I have made changes so that with minimum porting any other output formats could be incorporated. An additional getRecordWriter method needs to be implemented that takes an additional parameter. This parameter can be passed to fs.create (or even ignored as in the case of local filesystem.) > Progress in writing a DFS file does not count towards Job progress and can > make the task timeout > ------------------------------------------------------------------------------------------------ > > Key: HADOOP-318 > URL: http://issues.apache.org/jira/browse/HADOOP-318 > Project: Hadoop > Type: Bug > Components: mapred > Versions: 0.3.2 > Environment: all, but especially on big busy clusters > Reporter: Milind Bhandarkar > Assignee: Milind Bhandarkar > Fix For: 0.4.0 > Attachments: hadoop-latency-new.patch, hadoop-latency.patch > > When a task writes to DFS file, depending on how busy the cluster is, it can > timeout after 10 minutes by default, because the progress towards writing a > DFS file does not count as progress of the task. The solution (patch is > forthcoming) is to provide a way to callback reporter to report task progress > from DFSOutputStream. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira
