@JoeW, It looks like we need to add a ListS3 processor in addition to the Multipart Upload management that I'm looking into now. Extending ListFileTransfer for S3 shouldn't be too bad.
JoeS On Wed, Dec 2, 2015 at 12:04 AM, Joe Witt <joe.w...@gmail.com> wrote: > Hello > > So we have FetchS3 and PutHDFS and a series of interesting in between > processes to help. So that would get you most of the way there. How > to get the listing/know what to pull from S3? That part I'm not sure > about. > > This would make for a great example/template for us to post (as would > the reverse case). > > Thanks > Joe > > On Tue, Dec 1, 2015 at 10:36 PM, Naga Vijay <nagah...@gmail.com> wrote: > > Hello, > > > > Is there a processor to DistCp from Amazon S3 to HDFS, or do I need to > write > > a processor for it? > > > > Thanks > > Naga >
