[
https://issues.apache.org/jira/browse/OOZIE-426?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13899687#comment-13899687
]
Kevin Risden commented on OOZIE-426:
------------------------------------
Oozie supports any HCFS however the classes may not be present for Oozie to
work with any HCFS. An example of this is S3. Here are the steps needed to make
S3 work with Oozie at a high level:
1) Add S3 or S3N to the list of filesystems for
oozie.service.HadoopAccessorService.supported.filesystems
2) Add /usr/lib/hadoop/lib/jets3t-0.6.1.jar to the Oozie webserver lib directory
3) Restart Oozie
* you also need to specify your S3/S3N key but that would be standard across
Hadoop
The concrete code to do this on HDP 2 (and most likely others just not tested):
1) Add S3 or S3N to the list of filesystems for
oozie.service.HadoopAccessorService.supported.filesystems in Ambari
2) sudo cp /usr/lib/hadoop/lib/jets3t-0.6.1.jar
/var/lib/oozie/oozie-server/webapps/oozie/WEB-INF/lib/
3) sudo chown oozie:hadoop
/var/lib/oozie/oozie-server/webapps/oozie/WEB-INF/lib/jets3t-0.6.1.jar
4) Restart Oozie with Ambari
> OOZIE-9: Support (Amazon) s3/s3n schemes
> ----------------------------------------
>
> Key: OOZIE-426
> URL: https://issues.apache.org/jira/browse/OOZIE-426
> Project: Oozie
> Issue Type: New Feature
> Reporter: Hadoop QA
> Assignee: Roman Shaposhnik
>
> Currently Oozie supports only the 'hdfs' scheme. For people having their
> hadoop cluster in Amazon EC2 and using S3 for storage, it would be very
> useful if oozie supports the 's3n/s3' scheme(s). The use case I am talking
> about is as follows
> Hadoop cluster in Amazon EC2
> Uses hdfs for intermediate storage
> Uses s3 for getting input for, storing output of map-reduce jobs.
> More details on the above use-case and the exceptions/failures I have seen is
> documented here (http://tech.groups.yahoo.com/group/Oozie-users/message/1138).
> There can be other use cases as well - say use s3 as the DFS instead of HDFS.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
