[
https://issues.apache.org/jira/browse/NIFI-10112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Navneet Kaur updated NIFI-10112:
--------------------------------
Issue Type: Improvement (was: Wish)
> Error to slowdown for ListS3 and FetchS3
> ----------------------------------------
>
> Key: NIFI-10112
> URL: https://issues.apache.org/jira/browse/NIFI-10112
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Core Framework
> Affects Versions: 1.16.1
> Environment: Linux
> Reporter: Navneet Kaur
> Priority: Minor
>
> While listing and fetching objects from S3, and have received this error
> frequently:
> {color:#ff0000}{{ListS3 Failed to list contents of bucket due to Please
> reduce your request rate. (Service: Amazon S3; Status Code: 503; Error Code:
> Slowdown;.....)}}{color}
> {color:#ff0000}{{FetchS3 Failed to retrieve S3 object for FlowFile ...;
> routing to failure: com.amazonaws.services.s3.model.AmazonS3Exception: Please
> reduce your request rate. (Service: Amazon S3; Status Code: 503; Error Code:
> Slowdown;.....)}}{color}
>
> {color:#172b4d}Scenario: We were listing and fetching many files (1000+) on
> the buckets which are unique on the path, and was getting this error as
> transactions per second increased by 5.5k (I couldn't test if it actually
> increased by this number). Solution to this was setting "Run Schedule" time
> for ListS3 (5 mins) and FetchS3 (15sec).{color}
> {color:#172b4d}[https://aws.amazon.com/premiumsupport/knowledge-center/s3-resolve-503-slowdown-throttling/]{color}
> {color:#172b4d}
> This solution didn't solve the problem throughly, as I still get this error
> but sparsely.{color}
>
> {color:#172b4d}Looking forward to get this handled within the processors so
> that the transactions per second don't increase beyond the threshold.{color}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
