I also see a lot of blobStore-XXX directories in /tmp On Mon, Nov 10, 2014 at 8:48 AM, Ufuk Celebi <[email protected]> wrote:
> Hey Malte, > > thanks for reporting the issue. Did you change the default configuration? > I've just checked and the default config hard codes the heap size for the > task managers to 512 MB. In that case some of the algorithms will start > spilling to disk earlier than necessary (assuming that you have more main > memory available). You can either remove the config key " > taskmanager.heap.mb" and let the JVM set a default max heap size or set > the config key to a value appropriate for your machines/setup. Could you > try this out and report back? > > Regarding the temp directories: How much disk space is available before > running your program? I remember a user reporting the same issue, because > of some old files lingering around in the temp directories. > > – Ufuk > > > On Sun, Nov 9, 2014 at 10:54 PM, Malte Schwarzer <[email protected]> wrote: > >> Hi, >> >> I’m having the problem that my tmp dir is running out of hard drive >> space when running a map reduce job on a 1TB file. The job fails with "no >> space left on device“ exception. >> >> Probably the intermediate result set is getting too big. Is there a way >> to avoid this? Or make flink to write less temporary files? >> >> Thanks, >> Malte >> >> >> >
