Hello everyone, I am trying to upload a 300 MB file to azure blob storage using PutAzureBlobStorage and the processor is failing due to an OutOfMemoryError. My JVM heap size is set to 512 MB, but I wouldn't expect this to be an issue because the PutAzureBlobStorage processor should be using streaming to send the file to azure in chunks rather than reading it into memory in its entirety and then sending it out.
I am using NiFi version 1.12.1 Here's the error from the NiFi logs: > 2020-10-28 19:34:10,717 ERROR [Timer-Driven Process Thread-6] > o.a.n.p.a.storage.PutAzureBlobStorage > PutAzureBlobStorage[id=74b80a47-016d-3430-fd74-ece7653158d5] > PutAzureBlobStorage[id=74b80a47-016d-3430-fd74-ece7653158d5] failed to > process session due to java.lang.OutOfMemoryError: Java heap space; > Processor Administratively Yielded for 1 sec: java.lang.OutOfMemoryError: > Java heap space > java.lang.OutOfMemoryError: Java heap space > 2020-10-28 19:34:10,717 WARN [Timer-Driven Process Thread-6] > o.a.n.controller.tasks.ConnectableTask Administratively Yielding > PutAzureBlobStorage[id=74b80a47-016d-3430-fd74-ece7653158d5] due to > uncaught Exception: java.lang.OutOfMemoryError: Java heap space > java.lang.OutOfMemoryError: Java heap space > What's the recommendation for using NiFi to upload files of this size to blob storage? Thanks, Eric
