[
https://issues.apache.org/jira/browse/NIFI-16436?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
mathi updated NIFI-16436:
-------------------------
Description: {{ListS3}} in NiFi 2.2.0 appears to emit only a partial set of
objects from a configured S3 prefix when using {{{}Tracking Timestamps{}}}. A
test prefix contains around {*}1540 objects{*}, but only around *600 FlowFiles*
are emitted before the processor becomes idle. The downstream queue is empty,
and the processor remains running but does not emit additional FlowFiles for
several hours. Later, without configuration changes, it starts pulling files
again. This behavior is not dynamic/continuous and may indicate an issue with
timestamp tracking, listing batch progression, object age filtering,
S3-compatible storage pagination, or state checkpoint handling. Need
confirmation whether this is expected behavior or a potential defect. (was:
{{ListS3}} in NiFi 2.2.0 appears to emit only a partial set of objects from a
configured S3 prefix when using {{{}Tracking Timestamps{}}}. A test prefix
contains around 1540 objects, but only around 600 FlowFiles are emitted before
the processor becomes idle. The state checkpoint appears to advance, and
remaining objects are not emitted. Need confirmation whether this is expected
behavior or a potential issue with timestamp tracking, listing batch size, or
S3 pagination/state handling.)
> ListS3 with Tracking Timestamps lists only partial objects from S3 prefix and
> becomes idle before processing all existing objects
> ---------------------------------------------------------------------------------------------------------------------------------
>
> Key: NIFI-16436
> URL: https://issues.apache.org/jira/browse/NIFI-16436
> Project: Apache NiFi
> Issue Type: Bug
> Components: Tools and Build
> Affects Versions: 2.2.0
> Reporter: mathi
> Priority: Major
>
> {{ListS3}} in NiFi 2.2.0 appears to emit only a partial set of objects from a
> configured S3 prefix when using {{{}Tracking Timestamps{}}}. A test prefix
> contains around {*}1540 objects{*}, but only around *600 FlowFiles* are
> emitted before the processor becomes idle. The downstream queue is empty, and
> the processor remains running but does not emit additional FlowFiles for
> several hours. Later, without configuration changes, it starts pulling files
> again. This behavior is not dynamic/continuous and may indicate an issue with
> timestamp tracking, listing batch progression, object age filtering,
> S3-compatible storage pagination, or state checkpoint handling. Need
> confirmation whether this is expected behavior or a potential defect.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
