[
https://issues.apache.org/jira/browse/NIFI-14864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18014166#comment-18014166
]
Zenkovac commented on NIFI-14864:
---------------------------------
[~exceptionfactory]
this are my settings (same as nifi 1.19):
Processor config (scheduled every 5 secs):
Kafka Connection Service: my_kafka_controller
Group ID: my_consumer_group_id
Topic Format: names
Topics: mytopic
Auto Offset Reset: latest
Commit Offsets: true
Max Uncommitted Size: No value set (also tried 1MB, 10MB, etc with same effect)
Max Uncommitted Time: 10 secs
Header Name Pattern: No value set
Header Encoding: UTF-8
Processing Strategy: RECORD
Record Reader: JsonTreeReader
Record Writer: AvroRecordSetWriter
Output Strategy: Use Content as Value
Key Attribute Encoding: UTF-8 Encoded
Controller:
Bootstrap Servers: kafka1:6667, kafka1.temu.com.uy:6667, kafka1:6667
Security Protocol: PLAINTEXT
Transaction Isolation Level: Read Committed
Max Poll Records: 10000
Client Timeout: 60 sec
Max Metadata Wait Time: 5 sec
Acknowledgment Wait Time: 5 sec
Average sizes are always consistent ~500 records and ~260kb per flowfile.
Kafka is self-managed, 3 node cluster. (3 partitions and 3 replicas).
Im currently running nifi 1.19 and nifi 2.5 in paralell consuming from same
topic (with different consumer groups)
> ConsumeKafka performance
> ------------------------
>
> Key: NIFI-14864
> URL: https://issues.apache.org/jira/browse/NIFI-14864
> Project: Apache NiFi
> Issue Type: Bug
> Components: Configuration
> Affects Versions: 2.5.0
> Environment: nifi 2.5, kafka server 2.8
> Reporter: Zenkovac
> Priority: Major
>
> switching from nifi 1.19 to 2.5 and using ConsumeKafka cant get to consume
> flowfiles with more than ~500 records per flowfile despite having millions of
> messages available in kafka topic each flowfile
> This has a penalty performance for me because I consume thousands of
> flowfiles vs a few in nifi 1.19 hich means less disc i/o usage.
> this is my config:
> *Processing Strategy: RECORD*
> *Max Uncommitted Time* 10 sec
>
> despite having millions of messages available in kafka topic each flowfile
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
