Navneet Kaur created NIFI-10112:
-----------------------------------
Summary: Error to slowdown for ListS3 and FetchS3
Key: NIFI-10112
URL: https://issues.apache.org/jira/browse/NIFI-10112
Project: Apache NiFi
Issue Type: Wish
Components: Core Framework
Affects Versions: 1.16.1
Environment: Linux
Reporter: Navneet Kaur
While listing and fetching objects from S3, and have received this error
frequently:
{color:#FF0000}{{ListS3 Failed to list contents of bucket due to Please reduce
your request rate. (Service: Amazon S3; Status Code: 503; Error Code:
Slowdown;.....)}}{color}
{color:#FF0000}{{FetchS3 Failed to retrieve S3 object for FlowFile ...; routing
to failure: com.amazonaws.services.s3.model.AmazonS3Exception: Please reduce
your request rate. (Service: Amazon S3; Status Code: 503; Error Code:
Slowdown;.....)}}{color}
{color:#172b4d}Scenario: We were listing and fetching many files (1000+) on the
buckets which are unique on the path, and was getting this error as
transactions per second increased by 10k (I couldn't test if it actually
increased by this number). Solution to this was setting "Run Scheduler" time
for ListS3 (5 mins) and FetchS3 (15sec).
This solution didn't solve the problem throughly, as I still get this error but
sparsely.{color}
{color:#172b4d}Looking forward to get this handled within the processors so
that the transactions per second don't increase beyond the threshold.{color}
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
