[
https://issues.apache.org/jira/browse/NIFI-15526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18055303#comment-18055303
]
ASF subversion and git services commented on NIFI-15526:
--------------------------------------------------------
Commit 29d5040714982a57e9d1ca622f6e0479762d5309 in nifi's branch
refs/heads/main from Pierre Villard
[ https://gitbox.apache.org/repos/asf?p=nifi.git;h=29d5040714 ]
NIFI-15526 Added Fast Reader Enabled property to Avro Reader (#10829)
Signed-off-by: David Handermann <[email protected]>
> OOM triggered by AVRO org.apache.avro.fastread
> ----------------------------------------------
>
> Key: NIFI-15526
> URL: https://issues.apache.org/jira/browse/NIFI-15526
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
> Affects Versions: 2.7.1, 2.7.2
> Reporter: Josef Zahner
> Assignee: Pierre Villard
> Priority: Critical
> Attachments: image-2026-01-29-16-14-08-609.png,
> image-2026-01-29-16-14-29-698.png, image-2026-01-29-16-16-01-232.png
>
> Time Spent: 2.5h
> Remaining Estimate: 0h
>
> We found an issue causes OOM (used java heap full) for NiFi > v2.5.0 if we
> use AVRO Readers/Writers. It seems that the AVRO Library used in the newest
> NiFi versions has AVRO {{{}org.apache.avro.fastread{}}}=true per default.
> This causes massive OOM problems on our side. We suspect it is due to the
> inheritance of the reader/writer schemas.
> If we disable fastread in {{{}/conf/bootstrap.conf{}}}, the memory leak is
> gone:
> {code:java}
> java.arg.28=-Dorg.apache.avro.fastread=false{code}
> Here real world outputs based on prometheus metrics from used heap for NiFi.
> The screenshot 1 (blue line) shows a max 32GB heap, which never goes down
> again, it just grows (until it would reach 32GB and crashes). Thats taken
> with NiFI 2.7.2 with nothing changed regarding AVRO fast read (default, seems
> to be enabled).
> !image-2026-01-29-16-14-08-609.png!
> Screenshot 2 (orange line) shows another cluster with more or less the same
> use case, but with fastread=false. No memory leak anymore. So it's clearly
> related to the AVRO fastread feature.
> !image-2026-01-29-16-14-29-698.png!
> Yourkit Profiler example of the FastReaderBuilder object and memory usage. 1
> Object consumes about 10GB memory (fastread is on default, so enabled) .
> !image-2026-01-29-16-16-01-232.png!
>
> Would be great if someone of the devs, which is more familiar with all this
> topics, could investigate it.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
