[
https://issues.apache.org/jira/browse/NIFI-14864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18014634#comment-18014634
]
David Handermann commented on NIFI-14864:
-----------------------------------------
That sounds like a notable improvement with Kafka 3.
Using Run Once is not a great way to test ConsumeKafka because it goes through
a number of steps to create the Kafka Consumer, but those numbers are a helpful
point of comparison.
The FlowFile sizing depends on how many records it is able to pull from Kafka,
and this gets back to the max.partition.fetch.bytes property.
> ConsumeKafka performance
> ------------------------
>
> Key: NIFI-14864
> URL: https://issues.apache.org/jira/browse/NIFI-14864
> Project: Apache NiFi
> Issue Type: Bug
> Components: Configuration
> Affects Versions: 2.5.0
> Environment: nifi 2.5, kafka server 2.8
> Reporter: Zenkovac
> Priority: Major
>
> switching from nifi 1.19 to 2.5 and using ConsumeKafka cant get to consume
> flowfiles with more than ~500 records per flowfile despite having millions of
> messages available in kafka topic.
> This has a penalty performance for me because I consume thousands of
> flowfiles vs a few in nifi 1.19 which means less disc i/o usage.
> this is my config:
> *Processing Strategy: RECORD*
> *Max Uncommitted Time* 10 sec
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
