Hi Timo,
Yes, we are using off-heap memory, our yarn container are set to use ~23G
memory with two slot per container and set yarn heap cutoff ratio to 0.6.
Jobs are having normal memory usage, problem here is not temporary halt but
permanent halt for the running jobs.
Task manager's log
2018-0
Hi Amit,
how is the memory consumption when the jobs get stuck? Is the Java GC
active? Are you using off-heap memory?
Regards,
Timo
Am 2/12/18 um 10:10 AM schrieb Amit Jain:
Hi,
We have created Batch job where we are trying to merge set of S3
directories in TextFormat with the old snapshot
Hi,
We have created Batch job where we are trying to merge set of S3
directories in TextFormat with the old snapshot in Parquet format.
We are running 50 such jobs daily and found the progress of few random jobs
get stuck in between. We have gone through logs of JobManager, TaskManager
and could
