[
https://issues.apache.org/jira/browse/NIFI-14394?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dariusz Seweryn updated NIFI-14394:
-----------------------------------
Description:
ConsumeKinesisStream supports "Output Strategy" of "Use Wrapper" which wraps
message data and metadata into record content.
Currently metadata includes:
* stream
* shardId
* sequenceNumber
* partitionKey
* approximateArrival
What it does not include is Sub Sequence Number which is assigned to messages
that [are aggregated by the Amazon's Kinesis Producer
Library|https://docs.aws.amazon.com/streams/latest/dev/kinesis-producer-adv-aggregation.html]
making them indistinguishable by evaluating metadata.
Proposal is to add to wrapped metadata:
* subSequenceNumber — a long
* shardUniqueId — a String formed as `String.format("%s%020d", sequenceNumber,
subSequenceNumber)` which could be useful for uniquely identifying messages
identifying order using a single comparison
was:
ConsumeKinesisStream supports "Output Strategy" of "Use Wrapper" which wraps
message data and metadata into record content.
Currently metadata includes:
* stream
* shardId
* sequenceNumber
* partitionKey
* approximateArrival
What it does not include is Sub Sequence Number which is assigned to messages
that are aggregated by the Amazon's Kinesis Producer Library making them
indistinguishable by evaluating metadata.
Proposal is to add to wrapped metadata:
* subSequenceNumber — a long
*
shardUniqueId — a String formed as `String.format("%s%020d", sequenceNumber,
subSequenceNumber)` which could be useful for uniquely identifying messages
> ConsumeKinesisStream support for Sub Sequence Number
> ----------------------------------------------------
>
> Key: NIFI-14394
> URL: https://issues.apache.org/jira/browse/NIFI-14394
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Extensions
> Affects Versions: 2.3.0
> Reporter: Dariusz Seweryn
> Priority: Major
>
> ConsumeKinesisStream supports "Output Strategy" of "Use Wrapper" which wraps
> message data and metadata into record content.
> Currently metadata includes:
> * stream
> * shardId
> * sequenceNumber
> * partitionKey
> * approximateArrival
> What it does not include is Sub Sequence Number which is assigned to messages
> that [are aggregated by the Amazon's Kinesis Producer
> Library|https://docs.aws.amazon.com/streams/latest/dev/kinesis-producer-adv-aggregation.html]
> making them indistinguishable by evaluating metadata.
> Proposal is to add to wrapped metadata:
> * subSequenceNumber — a long
> * shardUniqueId — a String formed as `String.format("%s%020d",
> sequenceNumber, subSequenceNumber)` which could be useful for uniquely
> identifying messages identifying order using a single comparison
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
