[
https://issues.apache.org/jira/browse/HADOOP-619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12462581
]
Doug Cutting commented on HADOOP-619:
-------------------------------------
InputFormatBase#listPaths() no longer calls fs.makeQualified() on each path,
so, if a non-default FileSystem implementation does not return fully-qualified
paths (without scheme & authority) then things would be broken. Now, perhaps
it is a bug for a FileSystem#listPaths implementation ever to return a
non-fully-qualified path, but they don't all currently.
So, to be clear, the case I'm concerned about is if hdfs://xxx:1234/ is your
default fs, but you specify s3://yyy/zzz/*.txt as an input, and that glob
pattern expands to simply /yyy/foo.txt, then this file would not be found.
Does that make sense?
> Unify Map-Reduce and Streaming to take the same globbed input specification
> ---------------------------------------------------------------------------
>
> Key: HADOOP-619
> URL: https://issues.apache.org/jira/browse/HADOOP-619
> Project: Hadoop
> Issue Type: Improvement
> Components: mapred
> Affects Versions: 0.9.1
> Reporter: eric baldeschwieler
> Assigned To: Sanjay Dahiya
> Fix For: 0.10.0
>
> Attachments: Hadoop-619.patch, Hadoop-619.patch, Hadoop-619.patch,
> Hadoop-619_1.patch, Hadoop-619_1.patch, Hadoop-619_2.patch,
> Hadoop-619_2.patch, Hadoop-619_3.patch
>
>
> Right now streaming input is specified very differently from other map-reduce
> input. It would be good if these two apps could take much more similar input
> specs.
> In particular -input in streaming expects a file or glob pattern while MR
> takes a directory. It would be cool if both could take a glob patern of
> files and if both took a directory by default (with some patern excluded to
> allow logs, metadata and other framework output to be safely stored).
> We want to be sure that MR input is backward compatible over this change. I
> propose that a single file should be accepted as an input or a single
> directory. Globs should only match directories if the paterns is '/'
> terminated, to avoid massive inputs specified by mistake.
> Thoughts?
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
