[
https://issues.apache.org/jira/browse/NIFI-14820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18012173#comment-18012173
]
Joe Witt commented on NIFI-14820:
---------------------------------
[~awelless] I am thinking we should avoid adding options here. The expectation
is that ordering can be preserved and the requirement is that data is parseable
and the desire is we're performant.
Ordering preservation means we should kick out records in the ordering they
existed per topic partition.
Parseable means data that is together in batches with other data has a shared
schema.
Performant means we do utilize batches effectively so that we're not ending up
with flowfiles containing single records.
I'd advocate we don't need a new mode. But rather we simply clarify that for a
given topic/partition we will output flowfiles in batches that meet our
batching criteria and as long as the schema matches. If the schema changes
then we will definitely output a new flowfile.
> Add option to support original message ordering in ConsumeKafka
> ---------------------------------------------------------------
>
> Key: NIFI-14820
> URL: https://issues.apache.org/jira/browse/NIFI-14820
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Extensions
> Reporter: Alaksiej Ščarbaty
> Priority: Major
>
> When using schema registries it's possible that messages with different
> schemas land into the same topic. E.g. when doing a rolling update and
> updating the schema on producer, or simply writing different messages into
> the same topic.
> In several cases preserving message ordering (for a single partition) in a
> pipeline is important.
> For OLTP workloads messages can share casual dependencies, so they must be
> processed in a particular order. Or some downstream processors may rely on
> messages being in the exact order as they were received from Kafka.
> Currently {{ConsumeKafka}} groups records in flow files by their
> topic-partitions *as well as schemas*
> ([source|https://github.com/apache/nifi/blob/main/nifi-extension-bundles/nifi-kafka-bundle/nifi-kafka-processors/src/main/java/org/apache/nifi/kafka/processors/consumer/convert/AbstractRecordStreamKafkaMessageConverter.java#L128]).
> Which means the messages may be passed downstream out of order, due to the
> grouping.
> The processor should support both _Roll FlowFile_ (new) and _Group Records By
> Schema_ (existing) strategies, as in [ConsumeKinesis
> (WIP)|https://github.com/apache/nifi/pull/10053/files#diff-5e86a490a55dd29cce20638adb0f41401095aee8d94a734bf862c02f4ecf7fa8].
> To preserve backward compatibility, _Group Records By Schema_ should be
> chosen by default.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
