Hi everyone,
I am running spark jobs on EMR (using pyspark). I noticed that after
running jobs, the size of the usercache (basically the filecache folder)
keeps on increasing (with directory names as 1,2,3,4,5,...).
Directory location: */mnt/yarn/usercache/hadoop/**filecache/*
Is there a way to avoid creating these directories or automatically
clearing the usercache/filecache after a job/periodically?
--
--Thanks,
Shuporno Choudhury
- Clearing usercache on EMR [pyspark] Shuporno Choudhury
- Re: Clearing usercache on EMR [pyspark] Shuporno Choudhury
