[
https://issues.apache.org/jira/browse/NIFI-11811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17743402#comment-17743402
]
endzeit commented on NIFI-11811:
--------------------------------
By default the back pressure mechanisms of processors only affect the
scheduling. That is, if any of the downstream queues is full (according to its
configuration), the processor is no longer scheduled until the queue is not
full anymore.
However, once the processor is scheduled, it might put as many FlowFiles in the
downstream queues as it desires, even if the queue is deemed full based on its
back pressure configuration.
Let's say the downstream queue is limited to 10 FlowFiles and the following
processor is not started, thus not removing any FlowFiles from the queue. The
ListS3 processor is scheduled (as the queue is not full) and now searches the
defined S3 bucket. It might find 50 entries and will create a FlowFile for each
of them, even though the queue is set to only take up to 10. Afterwards the
queue is full however and the ListS3 processor will not be scheduled.
This is the default behaviour for all processors aligned with the core NiFi
framework. Additional work is required in the processor implementation to
actively watch for the downstream queues back pressure status. Almost no
processor does this, as far as I know.
It seems like your listing just contains a lot of objects. Unless the processor
is scheduled even though the queue is full, this sounds like a feature request
rather than a bug to me, e.g. introducing a "Max Batch Size" after which the
listing should be stopped. The existing "Listing Batch Size" only controls how
many FlowFoles will be commited in a single batch, but does not limit the
overall amount of FlowFiles to be emitted during a single scheduled run of the
processor.
> ListS3 Processor doesn't back pressure
> --------------------------------------
>
> Key: NIFI-11811
> URL: https://issues.apache.org/jira/browse/NIFI-11811
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
> Affects Versions: 1.16.2
> Reporter: Daniel Hermon
> Priority: Critical
>
> Dear NiFi team,
> I have built a data flow for syncing two S3 buckets on two separate sites.
> I am using NiFi because the throughput is much higher than any other tool
> (aws-s3-cli and RClone)
> The flow is pretty straightforward:
> ListS3 on-site A > FetchS3 from site A -> FetchS3 from site B (check if file
> already exists)> (on failure) PutS3Object in site B
> Sometimes site B throws 504 timeout exceptions which starts the back pressure
> mechanism.
> Everything back pressures perfectly fine but not the ListS3 processor output
> queue, I have to manually terminate it in order to stop listing new files,
> even trying to stop it fails.
> I have tried limiting the size (even though it is 0 bytes flow files since
> the flow file only contains attributes) and also limiting the number of
> objects.
>
> p.s the bucket has 5 Billion objects
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
