Hi all,
I am having some trouble processing 17 objects (total size 20.6GB ) through a
JsonEvaluateJsonPath processor.
Originally, the JVM had only 6GB and I progressively upgraded the amount of Ram
and it still fails with the following settings:
-Xms16g -Xmx40g -XX:MaxPermSize=6G -XX:PermSize=4G
The exact error message is:
2020-02-14 19:55:53,799 ERROR [Timer-Driven Process Thread-316]
o.a.n.p.standard.EvaluateJsonPath
EvaluateJsonPath[id=01701031-6cff-1beb-b313-ab0531781357]
EvaluateJsonPath[id=01701031-6cff-1beb-b313-ab0531781357] failed to process
session due to java.lang.OutOfMemoryError: Requested array size exceeds VM
limit; Pro
cessor Administratively Yielded for 1 sec: java.lang.OutOfMemoryError:
Requested array size exceeds VM limit
java.lang.OutOfMemoryError: Requested array size exceeds VM limit
at java.lang.StringCoding.encode(StringCoding.java:350)
at java.lang.String.getBytes(String.java:941)
at
org.apache.nifi.processors.standard.EvaluateJsonPath.lambda$onTrigger$3(EvaluateJsonPath.java:331)
at
org.apache.nifi.processors.standard.EvaluateJsonPath$$Lambda$840/682258977.process(Unknown
Source)
at
org.apache.nifi.controller.repository.StandardProcessSession.write(StandardProcessSession.java:2665)
at
org.apache.nifi.processors.standard.EvaluateJsonPath.onTrigger(EvaluateJsonPath.java:329)
at
org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
at
org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1162)
at
org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:205)
at
org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Are there any other settings I can tune? if not what are my options?
Thanks
