Hello Apache Hop users,
We often get this error in the DOS console despite (Java Heap Space):
2023/11/17 12:14:29 - Hop - Workflow execution has ended
Exception in thread "Timer-4" java.lang.OutOfMemoryError: Java heap
space
at java.base/java.util.Arrays.copyOf(Arrays.java:3745)
at
java.base/java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:120)
at
java.base/java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:95)
at
java.base/java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:156)
at
java.base/java.util.zip.DeflaterOutputStream.deflate(DeflaterOutputStream.java:262)
at
java.base/java.util.zip.DeflaterOutputStream.write(DeflaterOutputStream.java:211)
We have also observed that for a complex pipeline having >15000 rows in a json:
- It works well (whole pipeline could successful be executed) on the
developer machine (has >32MB RAM), this despite the fact that in the Dos
console you got the Java heap Space Error, Data looks also ok.
- It will never ends on a “standard” machine (has<12MB RAM), also on the
UI you wont find any error message (you might get in the DOS console the Java
Heap Space Error)
We played with the Row set size in the Meta-->Pipeline Run
Configuration-->Local but without success.
Kindly ask if there are good practices when we shall deal with "huge" json
(having >15000 rows/items)
Thank you + Cheers
Michel
