Wojciech Trefon created NIFI-16554:
--------------------------------------
Summary: Expose Kafka consumer fetch size properties in
Kafka3ConnectionService
Key: NIFI-16554
URL: https://issues.apache.org/jira/browse/NIFI-16554
Project: Apache NiFi
Issue Type: Improvement
Components: Extensions
Reporter: Wojciech Trefon
The Kafka consumer settings {{max.partition.fetch.bytes}} and
{{fetch.max.bytes}} control how much data a single poll can return, but
Kafka3ConnectionService (and AmazonMSKConnectionService, which extends it) only
supports them as dynamic properties, which users have to know about.
These settings matter most for ConsumeKafka with Schema Conflict Resolution set
to Continue with Merged Schema. In that mode each concurrent task holds all
records from a poll in memory until they are written. When consumers fall
behind, every poll is full, so heap usage grows with the number of tasks and
the fetch size. When each consumer is assigned only a few partitions, the
per-partition limit, not {{fetch.max.bytes}}, bounds the size of a poll. In a
test with a large backlog and several ConsumeKafka processors, lowering
{{max.partition.fetch.bytes}} from the default 1 MiB to 256 KiB eliminated
repeated full garbage collections and reduced peak heap usage, while lowering
{{fetch.max.bytes}} had no effect.
Proposal:
* Add {{max.partition.fetch.bytes}} and {{fetch.max.bytes}} as regular
properties of Kafka3ConnectionService, following the existing
{{max.poll.records}} property.
* Keep the Kafka client defaults (1048576 and 52428800 bytes), so existing
behavior does not change.
* Use the Kafka property names, so flows that already set these as dynamic
properties map onto the new properties without migration.
* Describe in the ConsumeKafka additional details how Continue with Merged
Schema relates to memory usage.
Related to NIFI-14271.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
