[
https://issues.apache.org/jira/browse/MAPREDUCE-1522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12836937#action_12836937
]
Tsz Wo (Nicholas), SZE commented on MAPREDUCE-1522:
---------------------------------------------------
A simple fix is to use Path.getFileSystem(conf).
{code}
diff --git src/mapred/org/apache/hadoop/mapreduce/lib/input/FileInputFormat.java
src/mapred/org/apache/hadoop/mapreduce/lib/input/FileInputFormat.java
index a2b6c49..04adeb0 100644
--- src/mapred/org/apache/hadoop/mapreduce/lib/input/FileInputFormat.java
+++ src/mapred/org/apache/hadoop/mapreduce/lib/input/FileInputFormat.java
@@ -354,7 +355,7 @@ public abstract class FileInputFormat<K, V> extends
InputFormat<K, V> {
public static void addInputPath(Job job,
Path path) throws IOException {
Configuration conf = job.getConfiguration();
- FileSystem fs = FileSystem.get(conf);
+ final FileSystem fs = path.getFileSystem(conf);
path = path.makeQualified(fs);
String dirStr = StringUtils.escapeString(path.toString());
String dirs = conf.get("mapred.input.dir");
{code}
We should also check the old org.apache.hadoop.mapred.FileInputFormat for this
problem.
> FileInputFormat may change the file system of an input path
> -----------------------------------------------------------
>
> Key: MAPREDUCE-1522
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1522
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: Tsz Wo (Nicholas), SZE
>
> org.apache.hadoop.mapreduce.lib.input.FileInputFormat.addInputPath(Job job,
> Path path) uses the default FileSystem but not the FileSystem specified in
> the path.
> {code}
> //org.apache.hadoop.mapreduce.lib.input.FileInputFormat
> public static void addInputPath(Job job,
> Path path) throws IOException {
> Configuration conf = job.getConfiguration();
> FileSystem fs = FileSystem.get(conf);
> path = path.makeQualified(fs); // the original FileSystem is lost.
> ...
> }
> {code}
> There is a similar problem in FileInputFormat.setInputPaths(..).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
