[
https://issues.apache.org/jira/browse/HADOOP-1558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alejandro Abdelnur updated HADOOP-1558:
---------------------------------------
Status: Patch Available (was: Open)
Resubmitting patch, added debug logging to OutputFormatBase temporary output
directory handling.
The testcase TestSymLink from contrib/streaming is failing.
It is failing because a build/env issue of the contrib testcases:
The contrib testcases use src/contrib/test/hadoop-site.xml.
The property 'mapred.system.dir' is this file is defined as with a variable
'${contrib.name}'.
The src/build/build-contrib.xml ant file sets the sysproperty 'contrib.name' to
the name of the contrib component for the JVM running the testcase.
The problem is that when a testcase uses MiniMRCluster the TaskRunner forks a
JVM for the task and in this JVM (which uses the above hadoop-site.xml) the
variable 'contrib.name' is undefined.
If I hardcode 'streaming' in the hadoop-site.xml for the TestSymLink the
testcase works fine.
> changes to OutputFormat to work on temporary directory to enable re-running
> crashed jobs (Issue: 1121)
> ------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-1558
> URL: https://issues.apache.org/jira/browse/HADOOP-1558
> Project: Hadoop
> Issue Type: Improvement
> Components: mapred
> Environment: all
> Reporter: Alejandro Abdelnur
> Fix For: 0.14.0
>
> Attachments: hadoop-1558-JUN1007-1934.txt
>
>
> Add OutputFormat methods like:
> /** Called to initialize output for this job. */
> void initialize(JobConf job) throws IOException;
> /** Called to finalize output for this job. */
> void commit(JobConf job) throws IOException;
> In the base implemenation for FileSystem output, initialize() might then
> create a temporary directory for the job, removing any that already exists,
> and commit could rename the temporary output directory to the final name.
> The existing checkOutputSpecs() would continue to throw an exception if the
> final output already exists.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
