Thanks Bryan, Joe, Adam and Pierre. I went past this issue by switching to
0.71. Now it is able to list the files from buckets and create those files
in the another bucket. But write is not happening and I am getting the
permission issue ( I have attached below for the reference) Could this be
the setting of the buckets or it has more to do with the access key. All
the files which are creaetd in the new bucket are of 0 byte.
Thanks
Rai
2016-10-28 16:45:25,438 ERROR [Timer-Driven Process Thread-3]
o.a.nifi.processors.aws.s3.FetchS3Object FetchS3Object[id=xxxxx] Failed to
retrieve S3 Object for
StandardFlowFileRecord[uuid=yyyyy,claim=,offset=0,name=xxxxx.gz,size=0];
routing to failure: com.amazonaws.services.s3.model.AmazonS3Exception:
Access Denied (Service: Amazon S3; Status Code: 403; Error Code:
AccessDenied; Request ID: xxxxxxx), S3 Extended Request ID:
lu8tAqRxu+ouinnVvJleHkUUyK6J6rIQCTw0G8G6DB6NOPGec0D1KB6cfUPsj08IQXI8idtiTp4=
2016-10-28 16:45:25,438 ERROR [Timer-Driven Process Thread-3]
o.a.nifi.processors.aws.s3.FetchS3Object
com.amazonaws.services.s3.model.AmazonS3Exception: Access Denied (Service:
Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID:
0F34E71C0697B1D8)
at
com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:1219)
~[aws-java-sdk-core-1.10.32.jar:na]
at
com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:803)
~[aws-java-sdk-core-1.10.32.jar:na]
at
com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:505)
~[aws-java-sdk-core-1.10.32.jar:na]
at
com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:317)
~[aws-java-sdk-core-1.10.32.jar:na]
at
com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3595)
~[aws-java-sdk-s3-1.10.32.jar:na]
at
com.amazonaws.services.s3.AmazonS3Client.getObject(AmazonS3Client.java:1116)
~[aws-java-sdk-s3-1.10.32.jar:na]
at
org.apache.nifi.processors.aws.s3.FetchS3Object.onTrigger(FetchS3Object.java:106)
~[nifi-aws-processors-0.7.1.jar:0.7.1]
at
org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
[nifi-api-0.7.1.jar:0.7.1]
at
org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1054)
[nifi-framework-core-0.7.1.jar:0.7.1]
at
org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:136)
[nifi-framework-core-0.7.1.jar:0.7.1]
at
org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
[nifi-framework-core-0.7.1.jar:0.7.1]
at
org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:127)
[nifi-framework-core-0.7.1.jar:0.7.1]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[na:1.8.0_101]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
[na:1.8.0_101]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
[na:1.8.0_101]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
[na:1.8.0_101]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[na:1.8.0_101]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[na:1.8.0_101]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_101]
On Fri, Oct 28, 2016 at 6:31 AM, Pierre Villard <pierre.villard...@gmail.com
> wrote:
> Quick remark: the fix has also been merged in master and will be in
> release 1.1.0.
>
> Pierre
>
> 2016-10-28 15:22 GMT+02:00 Gop Krr <gop....@gmail.com>:
>
>> Thanks Adam. I will try 0.7.1 and update the community on the outcome. If
>> it works then I can create a patch for 1.x
>> Thanks
>> Rai
>>
>> On Thu, Oct 27, 2016 at 7:41 PM, Adam Lamar <adamond...@gmail.com> wrote:
>>
>>> Hey All,
>>>
>>> I believe OP is running into a bug fixed here:
>>> https://issues.apache.org/jira/browse/NIFI-2631
>>>
>>> Basically, ListS3 attempts to commit all the files it finds
>>> (potentially 100k+) at once, rather than in batches. NIFI-2631
>>> addresses the issue. Looks like the fix is out in 0.7.1 but not yet in
>>> a 1.x release.
>>>
>>> Cheers,
>>> Adam
>>>
>>>
>>> On Thu, Oct 27, 2016 at 7:59 PM, Joe Witt <joe.w...@gmail.com> wrote:
>>> > Looking at this line [1] makes me think the FetchS3 processor is
>>> > properly streaming the bytes directly to the content repository.
>>> >
>>> > Looking at the screenshot showing nothing out of the ListS3 processor
>>> > makes me think the bucket has so many things in it that the processor
>>> > or associated library isn't handling that well and is just listing
>>> > everything with no mechanism of max buffer size. Krish please try
>>> > with the largest heap you can and let us know what you see.
>>> >
>>> > [1] https://github.com/apache/nifi/blob/master/nifi-nar-bundles/
>>> nifi-aws-bundle/nifi-aws-processors/src/main/java/org/apache
>>> /nifi/processors/aws/s3/FetchS3Object.java#L107
>>> >
>>> > On Thu, Oct 27, 2016 at 9:37 PM, Joe Witt <joe.w...@gmail.com> wrote:
>>> >> moving dev to bcc
>>> >>
>>> >> Yes I believe the issue here is that FetchS3 doesn't do chunked
>>> >> transfers and so is loading all into memory. I've not verified this
>>> >> in the code yet but it seems quite likely. Krish if you can verify
>>> >> that going with a larger heap gets you in the game can you please file
>>> >> a JIRA.
>>> >>
>>> >> Thanks
>>> >> Joe
>>> >>
>>> >> On Thu, Oct 27, 2016 at 9:34 PM, Bryan Bende <bbe...@gmail.com>
>>> wrote:
>>> >>> Hello,
>>> >>>
>>> >>> Are you running with all of the default settings?
>>> >>>
>>> >>> If so you would probably want to try increasing the memory settings
>>> in
>>> >>> conf/bootstrap.conf.
>>> >>>
>>> >>> They default to 512mb, you may want to try bumping it up to 1024mb.
>>> >>>
>>> >>> -Bryan
>>> >>>
>>> >>> On Thu, Oct 27, 2016 at 5:46 PM, Gop Krr <gop....@gmail.com> wrote:
>>> >>>>
>>> >>>> Hi All,
>>> >>>>
>>> >>>> I have very simple data flow, where I need to move s3 data from one
>>> bucket
>>> >>>> in one account to another bucket under another account. I have
>>> attached my
>>> >>>> processor configuration.
>>> >>>>
>>> >>>>
>>> >>>> 2016-10-27 20:09:57,626 ERROR [Flow Service Tasks Thread-2]
>>> >>>> org.apache.nifi.NiFi An Unknown Error Occurred in Thread
>>> Thread[Flow Service
>>> >>>> Tasks Thread-2,5,main]: java.lang.OutOfMemoryError: Java heap space
>>> >>>>
>>> >>>> I am very new to NiFi and trying ot get few of the use cases going.
>>> I need
>>> >>>> help from the community.
>>> >>>>
>>> >>>> Thanks again
>>> >>>>
>>> >>>> Rai
>>> >>>>
>>> >>>>
>>> >>>>
>>> >>>
>>>
>>
>>
>
