[
https://issues.apache.org/jira/browse/NIFI-14314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alex Leathwood updated NIFI-14314:
----------------------------------
Description:
In [NiFi-5213 |https://issues.apache.org/jira/browse/NIFI-5213]"Allow
AvroReader to process files w embedded schema even when the access strategy is
explicit schema" did so at the expense of [throwing and catching an
IOException|https://github.com/apache/nifi/blob/cd25603157556d28a4bda961ae5e52d18ac71849/nifi-extension-bundles/nifi-standard-services/nifi-record-serialization-services-bundle/nifi-record-serialization-services/src/main/java/org/apache/nifi/avro/AvroReaderWithExplicitSchema.java#L53]
for each and every Kafka record without an embedded schema, even when the
processor is correctly configured.
In high-performance use cases we have observed up to 2/3 of the cpu time ot the
respective worker thread being used for this check.
The exception originates for the kafka/avro client libraries and cannot easily
be replaced with a lightweight implementation of Exception .
We propose to either drop this earlier improvement of user experience (PR
available) )or to reimplement it so that the correct configuration is the
"happy path" and only the incorrect configuration case be penalized with
exceptions and ideally a warning.
was:
In NiFi-5213 "Allow AvroReader to process files w embedded schema even when the
access strategy is explicit schema" did so at the expense of [throwing and
catching an
IOException|https://github.com/apache/nifi/blob/cd25603157556d28a4bda961ae5e52d18ac71849/nifi-extension-bundles/nifi-standard-services/nifi-record-serialization-services-bundle/nifi-record-serialization-services/src/main/java/org/apache/nifi/avro/AvroReaderWithExplicitSchema.java#L53]
for each and every Kafka record without an embedded schema, even when the
processor is correctly configured.
In high-performance use cases we have observed up to 2/3 of the cpu time ot the
respective worker thread being used for this check.
The exception originates for the kafka/avro client libraries and cannot easily
be replaced with a lightweight implementation of Exception .
We propose to either drop this earlier improvement of user experience (PR
available) )or to reimplement it so that the correct configuration is the
"happy path" and only the incorrect configuration case be penalized with
exceptions and ideally a warning.
> Improve Kafka3 client performance when reading Avro
> ---------------------------------------------------
>
> Key: NIFI-14314
> URL: https://issues.apache.org/jira/browse/NIFI-14314
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Extensions
> Affects Versions: 2.2.0
> Reporter: Alex Leathwood
> Priority: Major
> Labels: performance, pull-request-available
>
> In [NiFi-5213 |https://issues.apache.org/jira/browse/NIFI-5213]"Allow
> AvroReader to process files w embedded schema even when the access strategy
> is explicit schema" did so at the expense of [throwing and catching an
> IOException|https://github.com/apache/nifi/blob/cd25603157556d28a4bda961ae5e52d18ac71849/nifi-extension-bundles/nifi-standard-services/nifi-record-serialization-services-bundle/nifi-record-serialization-services/src/main/java/org/apache/nifi/avro/AvroReaderWithExplicitSchema.java#L53]
> for each and every Kafka record without an embedded schema, even when the
> processor is correctly configured.
> In high-performance use cases we have observed up to 2/3 of the cpu time ot
> the respective worker thread being used for this check.
> The exception originates for the kafka/avro client libraries and cannot
> easily be replaced with a lightweight implementation of Exception .
> We propose to either drop this earlier improvement of user experience (PR
> available) )or to reimplement it so that the correct configuration is the
> "happy path" and only the incorrect configuration case be penalized with
> exceptions and ideally a warning.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
