Hi Andra, I would try increasing the memory per task manager, i.e. on a machine with 8 CPUs and 16GBs of memory, instead of spawning 8 TMs with 2GB each, I would try to spawn 2 TMs of 8GBs each. This might help with the spilling problem (in case that the CPU is not your bottleneck, this might even speed up the computations by avoiding spilling) and get you unstuck.
Cheers, Asterios On Fri, Jun 19, 2015 at 4:16 PM, Ufuk Celebi <u...@apache.org> wrote: > On 19 Jun 2015, at 14:53, Andra Lungu <lungu.an...@gmail.com> wrote: > > > Another problem that I encountered during the same set of > experiments(sorry > > if I am asking too many questions, I am eager to get things fixed): > > - for the same configuration, a piece of code runs perfectly on 10GB of > > input, then for 38GB it runs forever (no deadlock). > > > > I believe that may occur because Flink spills information to disk every > > time it runs out of memory... Is this fixable by increasing the number of > > buffers? > > If you are referring to the number of network buffers configuration key, > that should be unrelated. If this really is the issue, you can increase the > heap size for the task managers. > > Have you confirmed your suspicion as Till suggested via iotop? :) > > – Ufuk
