On 19 Jun 2015, at 14:53, Andra Lungu <lungu.an...@gmail.com> wrote: > Another problem that I encountered during the same set of experiments(sorry > if I am asking too many questions, I am eager to get things fixed): > - for the same configuration, a piece of code runs perfectly on 10GB of > input, then for 38GB it runs forever (no deadlock). > > I believe that may occur because Flink spills information to disk every > time it runs out of memory... Is this fixable by increasing the number of > buffers?
If you are referring to the number of network buffers configuration key, that should be unrelated. If this really is the issue, you can increase the heap size for the task managers. Have you confirmed your suspicion as Till suggested via iotop? :) – Ufuk
