My blobStore fileds are small, but each *.channel file is around 170MB. Before I start by Flink job I¹ve 25GB free space available in my tmp-dir and my taskmanager heap size is currently at 24GB. I¹m using a cluster with 10 nodes.
Is this enough space to process a 1TB file? Von: Stephan Ewen <[email protected]> Antworten an: <[email protected]> Datum: Montag, 10. November 2014 10:35 An: <[email protected]> Betreff: Re: How to make Flink to write less temporary files? I would assume that the blobStore fields are rather small (they are only jar files so far). I would look for *.channel files, which are spilled intermediate results. They can get pretty large for large jobs.
