[
https://issues.apache.org/jira/browse/NIFI-14882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18015528#comment-18015528
]
David Handermann commented on NIFI-14882:
-----------------------------------------
Commenting on here as noted on the initial draft PR:
There is certainly some overhead with record serialization and deserialization,
but any path forward should avoid being constrained to a particular format at
the Processor level. If there are reasons to consider something closer to the
Demarcator strategy, preserving the original format, that may be worth
considering as a new extension strategy with a Controller Service, versus
building support for specific formats in the Processor itself.
However, as Joe said, digging into the configuration details around the low
rate should be the next step.
> ConsumeKafka improvement for Avro+SchemaRegistry input
> ------------------------------------------------------
>
> Key: NIFI-14882
> URL: https://issues.apache.org/jira/browse/NIFI-14882
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Extensions
> Affects Versions: 2.5.0
> Environment: NiFi 2.4+, Kafka 3
> Reporter: Alex Goos
> Priority: Major
> Time Spent: 40m
> Remaining Estimate: 0h
>
> When Kafka receives Avro records marked with a schema identifier in the
> Confluent SchemaRegistry - and no immediate transformation is needed - the
> costly conversion into NiFiRecords and back to Avro is not needed. The
> performance can be substantially improved by simply batching records from the
> same source and with the same schema into a Avro Datafile.
> A new ProcessingStrategy "Avro Datafile" can be added to the existing ones.
> In our setup, a VM with a Intel(R) Xeon(R) CPU E5-2695 v4 @ 2.10GHz CPU, and
> a single executor thread for ConsumeKafka this bumps throughput from
> 500MB/5Min (Precessing Strategy RECORD) to 17GB/5min
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
