Zsihovszki Krisztina created NIFI-13487:
-------------------------------------------
Summary: ConsumeKinesisStream processor does not consume records
inserted before its start
Key: NIFI-13487
URL: https://issues.apache.org/jira/browse/NIFI-13487
Project: Apache NiFi
Issue Type: Bug
Affects Versions: 2.0.0-M4
Reporter: Zsihovszki Krisztina
Noticed a changed behaviour for ConsumeKinesisStream processor when
"Initial Stream Position" is set to "TRIM_HORIZON".
ConsumeKinesisStream consumes only records which were inserted to Kinesis
stream after the processor start and it does not consume the records which were
insterted before processor start.
In case of "TRIM_HORIZON" the records inserted before processor start (app
registration) should be consumed as well.
The processor behaved as it used "LATEST" initial stream position.
The issue occurs for NiFi 2.0 only and the root cause seems to be related to
NIFI-8531: Upgrading Kinesis processors to AWS SDK 2.x
According to
[https://docs.aws.amazon.com/streams/latest/dev/kcl-migration.html,] setting
initialPositionInStreamExtended was moved from LeaseManagementConfig to
RetrievalConfig.
The [existing
implementation|https://github.com/apache/nifi/blob/main/nifi-extension-bundles/nifi-aws-bundle/nifi-aws-processors/src/main/java/org/apache/nifi/processors/aws/kinesis/stream/ConsumeKinesisStream.java]
sets the value still in LeaseManagementConfig, this is the reason why the
default value, "LATEST" was used as initial position.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
