Yes, Falcon can be used to import / export from s3 and then subsequent
processesing can be orchestrated by falcon.
Ex: feed definition for s3 export/import (This ex: does an export to s3, by
switching the order an import can be realized)
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<feed name="some-feed-in-s3" description="some-feed-in-s3"
xmlns="uri:falcon:feed:0.1">
<frequency>hours(1)</frequency>
<timezone>UTC</timezone>
<late-arrival cut-off="hours(6)"/>
<clusters>
<cluster name="my-hdfs-cluster" type="source">
<validity start="2001-01-00T00:00Z" end="2999-12-31T00:00Z"/>
<retention limit="days(30)" action="delete"/>
</cluster>
<cluster name="my-s3-mirror" type="target">
<validity start="2001-01-00T00:00Z" end="2999-12-31T00:00Z"/>
<retention limit="days(90000)" action="delete"/>
<locations>
<location type="data"
path="s4://my-s3-bucket-name/feed-in-s3/${YEAR}-${MONTH}-${DAY}-${HOUR}/"/>
</locations>
</cluster>
</clusters>
<locations>
<location type="data"
path="/my/hdfs/path/feed-in-s3/${YEAR}/${MONTH}/${DAY}/${HOUR}/"/>
</locations>
<ACL owner="my-user" group="group" permission="0x755"/>
<schema location="/schema/mydata" provider="protobuf"/>
</feed>
To avoid exposing the S3 credentials in the s3-url, you can use the following
FS client and host the credentials in the user's home directory on hdfs.
https://github.com/InMobi/secure-s3-fsclient
Regards
Srikanth Sundarrajan
From: [email protected]
Date: Mon, 20 Jun 2016 11:07:22 +0200
Subject: Falcon and S3
To: [email protected]
Hi,
I'm a new to Falcon project, but I'm thinking to use it for workflow management
tool. I'm wondering if Falcon integrates with S3 well(i.e. export/import data)?
--
Best Regards,
Sayat Satybaldiyev
