Eric Secules created NIFI-7964:
----------------------------------
Summary: PutAzureBlobStorage OutOfMemory Exception
Key: NIFI-7964
URL: https://issues.apache.org/jira/browse/NIFI-7964
Project: Apache NiFi
Issue Type: Bug
Affects Versions: 1.12.1
Reporter: Eric Secules
As part of my flow I upload files to azure blob storage. They can be several
hundred MB in size. I putting a 300 MB file into my flow and it choked on the
PutAzureBlobStorage processor with the following log message.
{code:java}
2020-10-28 19:34:10,717 ERROR [Timer-Driven Process Thread-6]
o.a.n.p.a.storage.PutAzureBlobStorage
PutAzureBlobStorage[id=74b80a47-016d-3430-fd74-ece7653158d5]
PutAzureBlobStorage[id=74b80a47-016d-3430-fd74-ece7653158d5] failed to process
session due to java.lang.OutOfMemoryError: Java heap space; Processor
Administratively Yielded for 1 sec: java.lang.OutOfMemoryError: Java heap space
java.lang.OutOfMemoryError: Java heap space
2020-10-28 19:34:10,717 WARN [Timer-Driven Process Thread-6]
o.a.n.controller.tasks.ConnectableTask Administratively Yielding
PutAzureBlobStorage[id=74b80a47-016d-3430-fd74-ece7653158d5] due to uncaught
Exception: java.lang.OutOfMemoryError: Java heap space
java.lang.OutOfMemoryError: Java heap space
{code}
I did not expect this to happen because I think the PutAzureBlob processor
should be streaming the flowfile from disk directly to blob. But this behaviour
suggests to me that it's getting read into memory in its entirety. My JVM heap
size is set to 512 MB, which shouldn't be a problem if streaming was used to
upload to blob storage in chunks.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
