[
https://issues.apache.org/jira/browse/OOZIE-426?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13899764#comment-13899764
]
Robert Kanter commented on OOZIE-426:
-------------------------------------
{quote}All you really need is to have the FileSystem implementation somewhere
in lib/ , and the uri mapping is done for you by the FileSystem creation
API.{quote}
Correct. However, Oozie doesn't inherit Hadoop's classpath (launched jobs do,
but not the Oozie server itself). If we want the Oozie server to talk to s3 or
some other FileSystem, it needs the necessary jar in the WAR file (or
exploded/deployed WAR file). Oozie does everything through the FileSystem API,
so as long as the jar is there, it should work.
As I said before, I think all we need to do here is update the hadooplibs pom
files to make them include the jar with s3's FileSystem implementation in it.
If they ever add more FileSystem implementations in the future, then we'll have
to update the pom again if they're in another jar. In other words, for this
JIRA, the patch just needs to be a tweak to some pom files.
> OOZIE-9: Support (Amazon) s3/s3n schemes
> ----------------------------------------
>
> Key: OOZIE-426
> URL: https://issues.apache.org/jira/browse/OOZIE-426
> Project: Oozie
> Issue Type: New Feature
> Reporter: Hadoop QA
> Assignee: Roman Shaposhnik
>
> Currently Oozie supports only the 'hdfs' scheme. For people having their
> hadoop cluster in Amazon EC2 and using S3 for storage, it would be very
> useful if oozie supports the 's3n/s3' scheme(s). The use case I am talking
> about is as follows
> Hadoop cluster in Amazon EC2
> Uses hdfs for intermediate storage
> Uses s3 for getting input for, storing output of map-reduce jobs.
> More details on the above use-case and the exceptions/failures I have seen is
> documented here (http://tech.groups.yahoo.com/group/Oozie-users/message/1138).
> There can be other use cases as well - say use s3 as the DFS instead of HDFS.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
