[
https://issues.apache.org/jira/browse/NIFI-14864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18014166#comment-18014166
]
Zenkovac edited comment on NIFI-14864 at 8/15/25 5:25 PM:
----------------------------------------------------------
[~exceptionfactory]
this are my settings (same as nifi 1.19):
Processor config (scheduled every 5 secs):
Kafka Connection Service: my_kafka_controller
Group ID: my_consumer_group_id
Topic Format: names
Topics: mytopic
Auto Offset Reset: latest
Commit Offsets: true
Max Uncommitted Size: No value set (also tried 1MB, 10MB, etc with same effect)
Max Uncommitted Time: 10 secs
Header Name Pattern: No value set
Header Encoding: UTF-8
Processing Strategy: RECORD
Record Reader: JsonTreeReader
Record Writer: AvroRecordSetWriter
Output Strategy: Use Content as Value
Key Attribute Encoding: UTF-8 Encoded
Controller:
Bootstrap Servers: kafka1:6667, kafka1.temu.com.uy:6667, kafka1:6667
Security Protocol: PLAINTEXT
Transaction Isolation Level: Read Committed
Max Poll Records: 10000
Client Timeout: 60 sec
Max Metadata Wait Time: 5 sec
Acknowledgment Wait Time: 5 sec
Average sizes are always consistent ~500 records and ~260kb per flowfile.
Kafka is self-managed, 3 node cluster. (3 partitions and 3 replicas).
Im currently running nifi 1.19 and nifi 2.5 in paralell consuming from same
topic (with different consumer groups)
In version 1.19 changing "Max Uncommitted Time" is what controls how many
messages per flowfile i get since more time processing means more messages.
In 2.5 more time produces more small flowfiles.
I've also tried tunning "Max Uncommitted Size" with no effect.
was (Author: JIRAUSER294127):
[~exceptionfactory]
this are my settings (same as nifi 1.19):
Processor config (scheduled every 5 secs):
Kafka Connection Service: my_kafka_controller
Group ID: my_consumer_group_id
Topic Format: names
Topics: mytopic
Auto Offset Reset: latest
Commit Offsets: true
Max Uncommitted Size: No value set (also tried 1MB, 10MB, etc with same effect)
Max Uncommitted Time: 10 secs
Header Name Pattern: No value set
Header Encoding: UTF-8
Processing Strategy: RECORD
Record Reader: JsonTreeReader
Record Writer: AvroRecordSetWriter
Output Strategy: Use Content as Value
Key Attribute Encoding: UTF-8 Encoded
Controller:
Bootstrap Servers: kafka1:6667, kafka1.temu.com.uy:6667, kafka1:6667
Security Protocol: PLAINTEXT
Transaction Isolation Level: Read Committed
Max Poll Records: 10000
Client Timeout: 60 sec
Max Metadata Wait Time: 5 sec
Acknowledgment Wait Time: 5 sec
Average sizes are always consistent ~500 records and ~260kb per flowfile.
Kafka is self-managed, 3 node cluster. (3 partitions and 3 replicas).
Im currently running nifi 1.19 and nifi 2.5 in paralell consuming from same
topic (with different consumer groups)
In version 1.19 changing "Max Uncommitted Time" is what controls how many
messages per flowfile i get since more time processing means more messages.
In 2.5 more time produces more small flowfiles.
I've also tried tunning "Max Uncommitted Size" with no changes.
> ConsumeKafka performance
> ------------------------
>
> Key: NIFI-14864
> URL: https://issues.apache.org/jira/browse/NIFI-14864
> Project: Apache NiFi
> Issue Type: Bug
> Components: Configuration
> Affects Versions: 2.5.0
> Environment: nifi 2.5, kafka server 2.8
> Reporter: Zenkovac
> Priority: Major
>
> switching from nifi 1.19 to 2.5 and using ConsumeKafka cant get to consume
> flowfiles with more than ~500 records per flowfile despite having millions of
> messages available in kafka topic each flowfile
> This has a penalty performance for me because I consume thousands of
> flowfiles vs a few in nifi 1.19 hich means less disc i/o usage.
> this is my config:
> *Processing Strategy: RECORD*
> *Max Uncommitted Time* 10 sec
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
