Re: Memory manager behavior in iterative jobs

2016-01-30 Thread Fabian Hueske
Hi Gabor and Marton, the taskmanager.memory.preallocate switch basically replaces Flink's streaming mode. The current stream runtime code does not operate on managed memory. Hence, all memory allocated by the memory manager cannot be used for streaming jobs and is "lost". If the switch is set to

Memory manager behavior in iterative jobs

2016-01-30 Thread Gábor Gévay
Hello! We have a strangely behaving iterative Flink job: when we give it more memory, it gets much slower (more than 10 times). The problem seems to be mostly caused by GCs. Enabling object reuse didn’t help. With some profiling and debugging, we traced the problem to the operators requesting