[
https://issues.apache.org/jira/browse/NIFI-1542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15156178#comment-15156178
]
Joe Skora commented on NIFI-1542:
---------------------------------
[~tkurc], [~jameswing] My thoughts...
1. I don't like the idea of moving the age off call, in an environment with
occasional large uploads the age off wouldn't happen very often. Since state
age off is integrated with the S3 age off that might need to be rethought as
well. The age off is meant to be an occasional but consistent process and I
think the proposed #2 and #4 changes resolve the problem pretty nicely.
2. I agree with switching to a warn message, perhaps with a mention of the
ListUploads permission.
3. This can almost be done already with a very large age off interval, 1000
days for instance, but as I mentioned in #1 this will also affect local state
age off.
4. Great idea, I really like this especially in conjunction with #2. So, if
the S3 list uploads call fails with a permission error, a warn message will be
logged and the upload window advanced by the configured age off interval. This
will prevent checking S3 on every upload after a request fails due to
permissions, but allow it to try again on the next upload if the failure is for
any other reason.
I am starting work on the logging and interval reset changes. I should have a
pull request later today.
> PutS3Object Processor Logs Errors Without ListBucketMultipartUploads
> Permission
> -------------------------------------------------------------------------------
>
> Key: NIFI-1542
> URL: https://issues.apache.org/jira/browse/NIFI-1542
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
> Affects Versions: 0.5.0
> Environment: Linux, Chrome
> Reporter: James Wing
> Priority: Minor
> Fix For: 0.5.1
>
> Attachments: error-checking-s3-multipart-upload-list.png
>
>
> In 0.5.0, I am seeing an error putting objects to Amazon S3 using the
> PutS3Object processor. The *put operation itself succeeds*, but a nasty-gram
> is displayed in the flow and logged as a side effect. The error text is
> "Error checking S3 Multipart Upload list for <my s3 bucket>", from line 698
> of PutS3Object.java. This code was introduced in NIFI-1107, expanding the
> PutS3Object processor with support for S3 multi-part uploads of larger files.
> Logged in nifi-app.log:
> {code}
> 2016-02-19 20:56:24,312 ERROR [Timer-Driven Process Thread-3]
> o.a.nifi.processors.aws.s3.PutS3Object PutS
> 3Object[id=47df7533-91b2-4635-b865-45323f73f6c3] Error checking S3 Multipart
> Upload list for batchiq-scra
> tch: Access Denied (Service: Amazon S3; Status Code: 403; Error Code:
> AccessDenied; Request ID: 7483E76EA
> 52E33C4)
> 2016-02-19 20:56:24,533 INFO [Timer-Driven Process Thread-3]
> o.a.nifi.processors.aws.s3.PutS3Object PutS3
> Object[id=47df7533-91b2-4635-b865-45323f73f6c3] Successfully put
> StandardFlowFileRecord[uuid=a4cc19f5-cd2
> 4-4be7-8157-21f2609fa8cd,claim=StandardContentClaim
> [resourceClaim=StandardResourceClaim[id=1455912275117-1, container=default,
> section=1], offset=3072,
> length=1024],offset=0,name=273670477119454,size=1024] to Amazon S3 in 327
> milliseconds
> {code}
> The error-but-success behavior appears to happen because the exception is
> caught, logged, but not re-thrown. The approximate code path is as follows:
> # PutS3Object::onTrigger method calls ageoffS3Uploads regardless of
> single-part/multi-part code path
> # ageoffS3Uploads calls getS3AgeoffListAndAgeoffLocalState
> # getS3AgeoffListAndAgeoffLocalState calls s3.listMultipartUploads
> # After the exception, onTrigger proceeds with the single-part upload path
> *Troubleshooting Notes*
> Listing multi-part S3 uploads requires an additional S3 permission,
> s3:ListBucketMultipartUploads. I would expect the PutS3Object processor to
> only require the s3:PutObject permission for single-part uploads.
> I tested that this behavior did not occur in NiFi 0.4.2. I also tested that
> adding the s3:ListBucketMultipartUploads permission to my AWS credential
> circumvents the error messages from the S3PutObject processor.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
