Hey All, I believe OP is running into a bug fixed here: https://issues.apache.org/jira/browse/NIFI-2631
Basically, ListS3 attempts to commit all the files it finds (potentially 100k+) at once, rather than in batches. NIFI-2631 addresses the issue. Looks like the fix is out in 0.7.1 but not yet in a 1.x release. Cheers, Adam On Thu, Oct 27, 2016 at 7:59 PM, Joe Witt <joe.w...@gmail.com> wrote: > Looking at this line [1] makes me think the FetchS3 processor is > properly streaming the bytes directly to the content repository. > > Looking at the screenshot showing nothing out of the ListS3 processor > makes me think the bucket has so many things in it that the processor > or associated library isn't handling that well and is just listing > everything with no mechanism of max buffer size. Krish please try > with the largest heap you can and let us know what you see. > > [1] > https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-aws-bundle/nifi-aws-processors/src/main/java/org/apache/nifi/processors/aws/s3/FetchS3Object.java#L107 > > On Thu, Oct 27, 2016 at 9:37 PM, Joe Witt <joe.w...@gmail.com> wrote: >> moving dev to bcc >> >> Yes I believe the issue here is that FetchS3 doesn't do chunked >> transfers and so is loading all into memory. I've not verified this >> in the code yet but it seems quite likely. Krish if you can verify >> that going with a larger heap gets you in the game can you please file >> a JIRA. >> >> Thanks >> Joe >> >> On Thu, Oct 27, 2016 at 9:34 PM, Bryan Bende <bbe...@gmail.com> wrote: >>> Hello, >>> >>> Are you running with all of the default settings? >>> >>> If so you would probably want to try increasing the memory settings in >>> conf/bootstrap.conf. >>> >>> They default to 512mb, you may want to try bumping it up to 1024mb. >>> >>> -Bryan >>> >>> On Thu, Oct 27, 2016 at 5:46 PM, Gop Krr <gop....@gmail.com> wrote: >>>> >>>> Hi All, >>>> >>>> I have very simple data flow, where I need to move s3 data from one bucket >>>> in one account to another bucket under another account. I have attached my >>>> processor configuration. >>>> >>>> >>>> 2016-10-27 20:09:57,626 ERROR [Flow Service Tasks Thread-2] >>>> org.apache.nifi.NiFi An Unknown Error Occurred in Thread Thread[Flow >>>> Service >>>> Tasks Thread-2,5,main]: java.lang.OutOfMemoryError: Java heap space >>>> >>>> I am very new to NiFi and trying ot get few of the use cases going. I need >>>> help from the community. >>>> >>>> Thanks again >>>> >>>> Rai >>>> >>>> >>>> >>>
