[
https://issues.apache.org/jira/browse/OOZIE-426?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13899749#comment-13899749
]
jay vyas commented on OOZIE-426:
--------------------------------
Hmm... So im a little confused - why can't oozie just rely on hadoop for all
this and create generic FileSystem implementations from the configuration? Why
should it need to know anything about the filesystem? Isn't that what
"fs.default.name" is for?
All you really need is to have the FileSystem implementation somewhere in lib/
, and the uri mapping is done for you by the FileSystem creation API.
Please correct me if im off base here, or if im not appreciating something
complex that oozie does under the hood .
> OOZIE-9: Support (Amazon) s3/s3n schemes
> ----------------------------------------
>
> Key: OOZIE-426
> URL: https://issues.apache.org/jira/browse/OOZIE-426
> Project: Oozie
> Issue Type: New Feature
> Reporter: Hadoop QA
> Assignee: Roman Shaposhnik
>
> Currently Oozie supports only the 'hdfs' scheme. For people having their
> hadoop cluster in Amazon EC2 and using S3 for storage, it would be very
> useful if oozie supports the 's3n/s3' scheme(s). The use case I am talking
> about is as follows
> Hadoop cluster in Amazon EC2
> Uses hdfs for intermediate storage
> Uses s3 for getting input for, storing output of map-reduce jobs.
> More details on the above use-case and the exceptions/failures I have seen is
> documented here (http://tech.groups.yahoo.com/group/Oozie-users/message/1138).
> There can be other use cases as well - say use s3 as the DFS instead of HDFS.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
