[
https://issues.apache.org/jira/browse/NIFI-14882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18015220#comment-18015220
]
Alex Goos commented on NIFI-14882:
----------------------------------
Link to Draft PR: https://github.com/apache/nifi/pull/10222
> ConsumeKafka improvement for Avro+SchemaRegistry input
> ------------------------------------------------------
>
> Key: NIFI-14882
> URL: https://issues.apache.org/jira/browse/NIFI-14882
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Extensions
> Affects Versions: 2.5.0
> Environment: NiFi 2.4+, Kafka 3
> Reporter: Alex Goos
> Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> When Kafka receives Avro records marked with a schema identifier in the
> Confluent SchemaRegistry - and no immediate transformation is needed - the
> costly conversion into NiFiRecords and back to Avro is not needed. The
> performance can be substantially improved by simply batching records from the
> same source and with the same schema into a Avro Datafile.
> A new ProcessingStrategy "Avro Datafile" can be added to the existing ones.
> In our setup, a VM with a Intel(R) Xeon(R) CPU E5-2695 v4 @ 2.10GHz CPU, and
> a single executor thread for ConsumeKafka this bumps throughput from
> 500MB/5Min (Precessing Strategy RECORD) to 17GB/5min
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
