Alex Leathwood created NIFI-14314:
-------------------------------------
Summary: Improve Kafka3 client performance when reading Avro
Key: NIFI-14314
URL: https://issues.apache.org/jira/browse/NIFI-14314
Project: Apache NiFi
Issue Type: Improvement
Components: Extensions
Affects Versions: 2.2.0
Reporter: Alex Leathwood
In NiFi-5213 "Allow AvroReader to process files w embedded schema even when the
access strategy is explicit schema" did so at the expense of [throwing and
catching an
IOException|https://github.com/apache/nifi/blob/cd25603157556d28a4bda961ae5e52d18ac71849/nifi-extension-bundles/nifi-standard-services/nifi-record-serialization-services-bundle/nifi-record-serialization-services/src/main/java/org/apache/nifi/avro/AvroReaderWithExplicitSchema.java#L53]
for each and every Kafka record without an embedded schema, even when the
processor is correctly configured.
In high-performance use cases we have observed up to 2/3 of the cpu time ot the
respective worker thread being used for this check.
The exception originates for the kafka/avro client libraries and cannot easily
be replaced with a lightweight implementation of Exception .
We propose to either drop this earlier improvement of user experience (PR
available) )or to reimplement it so that the correct configuration is the
"happy path" and only the incorrect configuration case be penalized with
exceptions and ideally a warning.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
