[
https://issues.apache.org/jira/browse/NIFI-14882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18021066#comment-18021066
]
Alex Goos commented on NIFI-14882:
----------------------------------
[~joewitt] :
The rate is not particularly low, given that we deliberatly just ran one Thread
for the Consumer ofr measuring the impact. In production we have many more
threads configured for this, but would like not to over-spend either.
Please have a look at the attached screenshot from the Profiler. We interpret
it as such as the processor spends 72% of the CPU time only in the
"createRecordReader" method.
Overall 96% is spent in "AbstractRecordStreamKafkaMessageConverter.toFlowFiles".
In the very common case of using Confluence SchemaRegistry to encode
AvroSchemas for one AvroBinary Message per KafkaMessage and then continuing to
pass the messages in Avro format inside the NiFi Cluster, this is pure oerhead.
@[~exceptionfactory] :
Using the demarcator strategy would mean we'd have to inspect the bytestream
and re-shuffle per Schema identifier at a later custom processor stage.
And since the internal structure of the ConsumeKafka processor is neatly
divided in one class per ProcessingStrategy, we'd like this low-invasive code
with the community.
I don't see how this would fit into a ControllerService. It's just a matter of
batching the results of the KafkaClient ControllerService differently.
!Screenshot from 2025-08-27 13-52-01.png!
> ConsumeKafka improvement for Avro+SchemaRegistry input
> ------------------------------------------------------
>
> Key: NIFI-14882
> URL: https://issues.apache.org/jira/browse/NIFI-14882
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Extensions
> Affects Versions: 2.5.0
> Environment: NiFi 2.4+, Kafka 3
> Reporter: Alex Goos
> Priority: Major
> Attachments: Screenshot from 2025-08-27 13-52-01.png
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> When Kafka receives Avro records marked with a schema identifier in the
> Confluent SchemaRegistry - and no immediate transformation is needed - the
> costly conversion into NiFiRecords and back to Avro is not needed. The
> performance can be substantially improved by simply batching records from the
> same source and with the same schema into a Avro Datafile.
> A new ProcessingStrategy "Avro Datafile" can be added to the existing ones.
> In our setup, a VM with a Intel(R) Xeon(R) CPU E5-2695 v4 @ 2.10GHz CPU, and
> a single executor thread for ConsumeKafka this bumps throughput from
> 500MB/5Min (Precessing Strategy RECORD) to 17GB/5min
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
