[
https://issues.apache.org/jira/browse/NIFI-14864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18014164#comment-18014164
]
David Handermann commented on NIFI-14864:
-----------------------------------------
Can you provide the additional details from the Kafka Connection Service,
including any custom properties configured?
Output FlowFile size and number of records per FlowFile depend on several
different Kafka properties, which may be one factor to consider.
It would also be helpful to know the average size of each record.
ConsumeKafka is heavily used, and I have observed much higher rates.
The Environment mentions Kafka 2.8, is this a particular version from a
specific provider, or running on self-managed servers?
> ConsumeKafka performance
> ------------------------
>
> Key: NIFI-14864
> URL: https://issues.apache.org/jira/browse/NIFI-14864
> Project: Apache NiFi
> Issue Type: Bug
> Components: Configuration
> Affects Versions: 2.5.0
> Environment: nifi 2.5, kafka server 2.8
> Reporter: Zenkovac
> Priority: Major
>
> switching from nifi 1.19 to 2.5 and using ConsumeKafka cant get to consume
> flowfiles with more than ~500 records per flowfile despite having millions of
> messages available in kafka topic each flowfile
> This has a penalty performance for me because I consume thousands of
> flowfiles vs a few in nifi 1.19 hich means less disc i/o usage.
> this is my config:
> *Processing Strategy: RECORD*
> *Max Uncommitted Time* 10 sec
>
> despite having millions of messages available in kafka topic each flowfile
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
