Hello The code shown is very simple but not memory efficient.
The first call IOUtils.toString takes a 'stream' and converts it into a full java String object in memory. So in a simple sense if the input is 190MB then you have at least that in the Java Heap. Then again the call text text.GetBytes would do the same even if temporarily. You want to do these changes in batches or using mechanisms that allow it to happen in streaming fashion. There are a lot of parts of NiFi that do such things to ensure efficient memory usage. Others more familiar with Groovy/etc.. can certainly provide pointers. Thanks On Thu, Nov 14, 2024 at 7:03 AM <e-soci...@gmx.fr> wrote: > Hello all, > > Why I got out of memory during processing the file with > ExecuteGroovyScript? > > The size of file could be reach a maxsize to 190Mo > > Error : ExecuteGroovyScript[id=07e5314d-b20a-1076-882b-54b44baca66d] > java.lang.OutOfMemoryError: Java heap space: java.lang.OutOfMemoryError: Java > heap space > > The groovy is very simple : > > // get data from flowfile > text = IOUtils.toString(inputStream, StandardCharsets.UTF_8) > > // add timestamp in each line in the text/flowfile > text = text.replaceAll(/^/, "\"timestamp\":\"$timestamp\",") > > // write back to the flowfile > outputStream.write(text.getBytes(StandardCharsets.UTF_8)) > > Could you help me to process if it is not the good way to do this thing ? > > Thanks > > Minh > > >
