[
https://issues.apache.org/jira/browse/NIFI-14864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18014195#comment-18014195
]
David Handermann commented on NIFI-14864:
-----------------------------------------
Thanks for the environment details, that is helpful.
Have you tried with a newer version of Kafka?
The new Kafka components are built and tested against Kafka 3 and 4, I haven't
tried them with Kafka 2.
It could be worth experimenting with the *fetch.max.bytes* and
*max.partition.fetch.bytes* Kafka properties, which can be set as custom
properties in the Kafka Connection Service. If individual FlowFiles are only
260 KB, that sounds very small, and setting these values to something like 1 MB
or 2 MB (with the exact byte value in the case of those properties) would be
interesting.
> ConsumeKafka performance
> ------------------------
>
> Key: NIFI-14864
> URL: https://issues.apache.org/jira/browse/NIFI-14864
> Project: Apache NiFi
> Issue Type: Bug
> Components: Configuration
> Affects Versions: 2.5.0
> Environment: nifi 2.5, kafka server 2.8
> Reporter: Zenkovac
> Priority: Major
>
> switching from nifi 1.19 to 2.5 and using ConsumeKafka cant get to consume
> flowfiles with more than ~500 records per flowfile despite having millions of
> messages available in kafka topic.
> This has a penalty performance for me because I consume thousands of
> flowfiles vs a few in nifi 1.19 which means less disc i/o usage.
> this is my config:
> *Processing Strategy: RECORD*
> *Max Uncommitted Time* 10 sec
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
