[
https://issues.apache.org/jira/browse/NIFI-15526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Josef Zahner updated NIFI-15526:
--------------------------------
Description:
We found an issue causes OOM (used java heap full) for NiFi > v2.5.0 if we use
AVRO Readers/Writers. It seems that the AVRO Library used in the newest NiFi
versions has AVRO {{{}org.apache.avro.fastread{}}}=true per default. This
causes massive OOM problems on our side. We suspect it is due to the
inheritance of the reader/writer schemas.
If we disable fastread in {{{}/conf/bootstrap.conf{}}}, the memory leak is gone:
{code:java}
java.arg.28=-Dorg.apache.avro.fastread=false{code}
Here real world outputs based on prometheus metrics from used heap for NiFi.
The screenshot 1 (blue line) shows a max 32GB heap, which never goes down
again, it just grows (until it would reach 32GB and crashes). Thats taken with
NiFI 2.7.2 with nothing changed regarding AVRO fast read (default, seems to be
enabled).
!image-2026-01-29-16-14-08-609.png!
Screenshot 2 (orange line) shows another cluster with more or less the same use
case, but with fastread=false. No memory leak anymore. So it's clearly related
to the AVRO fastread feature.
!image-2026-01-29-16-14-29-698.png!
Yourkit Profiler example of the FastReaderBuilder object and memory usage. 1
Object consumes about 10GB memory (fastread is on default, so enabled) .
!image-2026-01-29-16-16-01-232.png!
Would be great of someone of the devs is more familiar with all this topics
could investigate it.
was:
We found an issue causes OOM (used java heap full) for NiFi > v2.5.0 if we use
AVRO Readers/Writers. It seems that the AVRO Library used in the newest NiFi
versions has AVRO {{{}org.apache.avro.fastread{}}}=true per default. This
causes massive OOM problems on our side. We suspect it is due to the
inheritance of the reader/writer schemas.
If we disable fastread in {{{}/conf/bootstrap.conf{}}}, the memory leak is gone:
{{}}
{code:java}
java.arg.28=-Dorg.apache.avro.fastread=false{code}
{{}}
Here real world outputs based on prometheus metrics from used heap for NiFi.
The screenshot 1 (blue line) shows a max 32GB heap, which never goes down
again, it just grows (until it would reach 32GB and crashes). Thats taken with
NiFI 2.7.2 with nothing changed regarding AVRO fast read (default, seems to be
enabled).
!image-2026-01-29-16-14-08-609.png!
Screenshot 2 (orange line) shows another cluster with more or less the same use
case, but with fastread=false. No memory leak anymore. So it's clearly related
to the AVRO fastread feature.
!image-2026-01-29-16-14-29-698.png!
Yourkit Profiler example of the FastReaderBuilder object and memory usage. 1
Object consumes about 10GB memory (fastread is on default, so enabled) .
!image-2026-01-29-16-16-01-232.png!
Would be great of someone of the devs is more familiar with all this topics
could investigate it.
> OOM triggered by AVRO org.apache.avro.fastread
> ----------------------------------------------
>
> Key: NIFI-15526
> URL: https://issues.apache.org/jira/browse/NIFI-15526
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
> Affects Versions: 2.7.1, 2.7.2
> Reporter: Josef Zahner
> Priority: Critical
> Attachments: image-2026-01-29-16-14-08-609.png,
> image-2026-01-29-16-14-29-698.png, image-2026-01-29-16-16-01-232.png
>
>
> We found an issue causes OOM (used java heap full) for NiFi > v2.5.0 if we
> use AVRO Readers/Writers. It seems that the AVRO Library used in the newest
> NiFi versions has AVRO {{{}org.apache.avro.fastread{}}}=true per default.
> This causes massive OOM problems on our side. We suspect it is due to the
> inheritance of the reader/writer schemas.
> If we disable fastread in {{{}/conf/bootstrap.conf{}}}, the memory leak is
> gone:
> {code:java}
> java.arg.28=-Dorg.apache.avro.fastread=false{code}
> Here real world outputs based on prometheus metrics from used heap for NiFi.
> The screenshot 1 (blue line) shows a max 32GB heap, which never goes down
> again, it just grows (until it would reach 32GB and crashes). Thats taken
> with NiFI 2.7.2 with nothing changed regarding AVRO fast read (default, seems
> to be enabled).
> !image-2026-01-29-16-14-08-609.png!
> Screenshot 2 (orange line) shows another cluster with more or less the same
> use case, but with fastread=false. No memory leak anymore. So it's clearly
> related to the AVRO fastread feature.
> !image-2026-01-29-16-14-29-698.png!
> Yourkit Profiler example of the FastReaderBuilder object and memory usage. 1
> Object consumes about 10GB memory (fastread is on default, so enabled) .
> !image-2026-01-29-16-16-01-232.png!
>
> Would be great of someone of the devs is more familiar with all this topics
> could investigate it.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
